Learn how to use IBM® App Connect Enterprise on IBM Cloud to write or transfer files in the directories available on an SFTP Server over the public internet. This SFTP server is accessible using the username and password option.
Your company wants to process business data that is stored in files in a number of remote file locations that are exposed using the SSH File Transfer Protocol (SFTP). This protocol is used as the reading and writing of file data is taking place over the public internet.
An integration solution needs to be constructed that securely reads the files from various directories, processes the data and writes data back to the same the remote locations. In this tutorial an existing integration solution is imported to run in IBM App Connect Enterprise on IBM Cloud to do such processing.
The file transport used is SFTP, which is secure as it runs on top of the SSH protocol. This is important as the communication between the integration running in IBM App Connect Enterprise on IBM Cloud and the SFTP server is across the public internet and should not be in plain text.
Authentication with the SFTP server is done using username and password.
In those cases where the SFTP server is not accessible on the public internet then a secure gateway facility can be used to provide a secure access between the different networks. This is discussed at the end of this tutorial.
This scenario presents one type of usage but it can easily be extended to meet your own needs.
First, find or create everything you need:
- An SFTP Server available over the public internet that has multiple remote directories.
You’ll need the following information, that will be provided by SFTP Server Administrator:
- Host name of the server
- Port used by server for TCP/IP communication; for example, 22
- Directory locations for reading and writing the files
- SFTP Server Username
- SFTP Server Password
- An IBM Cloud account with an IBM App Connect service that provides enterprise integration capabilities.
- Optionally, you can also install IBM App Connect Enterprise for Developers on your local development environment if you want to look at and change the deployed message flow. It is not necessary for running the tutorial, but it is useful for examining the message flow to understand how it works.
(Note: The integration project for this tutorial was developed with the IBM App Connect Enterprise Toolkit).
- If you want a bit more information before you start, you can read more about the IBM App Connect Enterprise for Developers edition on the following page: Get started with IBM App Connect Enterprise.
Import the integration project into the IBM App Connect Enterprise Toolkit to look at the message flow
You do not need to complete this step unless you want to look at or change the integration service. A separate BAR file is also provided for deploying the integration into IBM App Connect Enterprise on IBM Cloud, as described in a later step.
All the integration service resources required for this tutorial are provided in a project interchange file named CustomerRecords-SFTP.zip and this file is located within the zip file CustomerRecords-SFTP-and-data.zip. So extract CustomerRecords-SFTP.zip from the attachment before attempting to import the project interchange file.
The steps to import the integration service into the IBM App Connect Enterprise Toolkit are given below:
- Download the project interchange file by clicking the link above and saving the file to a local directory.
- Open the IBM App Connect Enterprise Toolkit.
- Import the project into the toolkit by clicking File > Import. Expand IBM Integration, select Project Interchange, and then click Next.
- Browse to select the downloaded project interchange file, and then click Finish.
- A new application called CustomerRecordsApp, is displayed in the IBM App Connect Enterprise Toolkit:
The Application has one message flow called CustomerRecords.mgflow that implements custom logic. The message flow performs the following actions:
The message flow polls files from three different source directories of the SFTP Server using three different FileInputNodes (here named as JSON Data, XML Data, DFDL Data). It is assumed that the same logical data is uploaded into the source directories in three different formats (JSON, XML and TXT format) that correspond with the node names.
When a file is identified in any of these source directories, the message flow processes the records present in the file, and then based on the customer status (Platinum, Gold, Silver) it generates three different files (all in txt format) that are written to different target directories (Platinum, Gold or Silver) using multiple FileOutputNodes
The directory names that are configured on the FileInput and FileOutput nodes on the Basic tab must start with a prefix of /tmp. This is a requirement that is specific to IBM App Connect Enterprise on IBM Cloud and it is because of the way the feature is implemented, and locations available, when running in the managed service.
You will also need to specify the same directory structure later when you will create an SFTP Policy in the IBM App Connect Enterprise on IBM Cloud instance.
In the same policy you will also be required to provide the same Security identity name that is specified in the Security identity box in the FTP Tab for the FileInput node. For example, DummyIdentity in this case.
The following images show the properties on the Basic and FTP tabs for the JSON Data, XML Data and DFDL Data FileInputNodes.
JSON Data Basic Tab:
XML Data Basic Tab:
DFDL Data Basic Tab:
JSON Data FTP Tab:
XML Data FTP Tab:
DFDL Data FTP Tab:
The following images show the properties from the Basic and FTP tab for the Platinum, Gold and Silver FileOutputNodes.
Platinum Basic Tab:
Gold Basic Tab:
Silver Basic Tab:
Platinum FTP Tab:
Gold FTP Tab:
Silver FTP Tab:
Configure the integration in IBM App Connect Enterprise on IBM Cloud:
- Download the zip file CustomerRecords-SFTP-and-data.zip and extract the BAR file named CustomerRecords.bar from it.
- If necessary, sign in to IBM Cloud. From the IBM Cloud dashboard, select and then launch your App Connect service instance.
- From the App Connect on IBM Cloud dashboard, click New > Import a BAR file and select the BAR file that you extracted and/or changed. Then click Import.
The integration server that is created as a result of the import is displayed in the dashboard.
It is now necessary to create an SFTP policy so that property overrides can be specified.
- From the IBM App Connect Enterprise on IBM Cloud menu, click , then Manage > Policies, to open the Policies view.
- Click New policy.
- Create an SFTP(Security Identity) policy with the following details:
- Policy name: Enter
DummyPolicy. You can change it to any name based on naming conventions and standards.
- Security Identity: This should be the same value that is specified in the Security identity box in the FTP Tab of all FileInputNode and FileOutputNode instances in this message flow. For example, In this case
DummyIdentityto match with the values shown in the screen shots of the node properties, that were shown in the previous section.
- User name: This is the SFTP Server user-name that the message flow will use to identify with to the SFTP server.
- Password: This is the password for the message flow to login to the SFTP Server.
- Directories to create: This is the list of the names of directories that need to be created as part of the configuration when this message flow is run. All directories must have a prefix of /tmp and this will be automatically added to the entered text as soon as the ‘Add’ icon is clicked. The directory names entered need to match the values supplied for all of the:
• “Input Directory” values in the Basic tab of all the FileInputNode instances
• Directory values in the Basic tab of all of the FileOutputNode instances
You can add more than one directory structure to the policy if required. For example, here the following directories will be added/created while creating the policy
- Policy name: Enter
- Click Create and then return to the IBM App Connect Enterprise on IBM Cloud dashboard.
- To attach the policy to the uploaded BAR file, search for the policy name in the search bar and select ‘Attach policy’ from the actions.
- In the pop-up window check the ‘CustomerRecords’ Integration Server and click Apply.
- Once selected, the value for ‘Used by’ will change from ‘0 integration servers to ‘1 integration server.
- Return to the dashboard. Then start the integration by opening the options menu [⋮] for the integration server, and then clicking Start. When the integration shows Running, the integration is running and ready to use.
Finally, test your integration:
To test the integration you will copy the sample data that is provided into the directories that the message flow is monitoring and then after a short time you will look in the output directories to verify the files were written.
- Login to the SFTP Server using the provided credentials.
- Copy the sample data files (customer.json, customer.txt and customer.xml) that are in the zip file CustomerRecords-SFTP-and-data.zip into the corresponding SFTP server directories.
- Wait a short time for the files to be read from the directories. Then check if the files have been read from all the input directories.
- Check the Output Directories to see if the data has been written to files as per the logic in the Application
- Go to the directory named in the FTP tab of the Platinum FileOutputNode and check the contents of the file. The output should be similar to this.
- Go to the directory named in the FTP tab of the Gold FileOutputNode and check the contents of the file. The output should be similar to this.
- Go to the directory named in the FTP tab of the Silver FileOutputNode and check the contents of the file. The output should be similar to this.
If all of these steps work, then you have the tutorial up and running successfully. If it fails, then an error message with the reason will be returned. Check the integration log for any errors – you can access the logs from the ’Manage > Logs’ menu item in the dashboard.
Connecting to SFTP Server using IBM Secure Gateway:
If the SFTP Server is hosted on-premises or in a private cloud (i.e. not accessible using the public internet), and you do not want to open a separate port on the firewall, then you need a service that is able to provide secure access between the different networks. IBM Secure Gateway Service is such a service. It provides connectivity to on-premise private networks or environments available in private cloud and it can be used to securely access the SFTP Server. You can read more about the Secure Gateway on the following page: Getting started with IBM Secure Gateway.
This diagram shows the configuration that you would use when using IBM Secure Gateway Service
Secure Gateway Service Configuration
From the IBM Cloud, complete the following step to configure the Secure Gateway service to connect to an SFTP Server over the public internet:
- Create one gateway in an IBM Secure Gateway service instance provisioned in IBM Cloud.
- Ask the SFTP Administrator to install the Secure Gateway Client in the SFTP server environment. See the section Secure Gateway Client Installation and Configuration below to see more details on this task.
- Share the gateway id and Security token with SFTP Administrator to configure it in the IBM Secure Gateway client. Once configured, the gateway will be connected to the client using the provided gateway ID.
- Ask the SFTP Administrator to provide the SFTP Server host name/IP and SFTP server port detail, that will be configured in the destination.
- Create a destination in the IBM Secure Gateway service instance with the SFTP server host and SFTP server port provided by the SFTP Administrator.
- In the case that an ‘Access is blocked’ sign appears on the destination ask the SFTP Administrator to add the SFTP server host and SFTP server port entry in the ACL to allow the communication.
The next step is to configure the Secure Gateway client
Secure Gateway Client Installation and Configuration
The Secure Gateway client needs to be installed on a system that is adjacent to the SFTP server. This would most likely be on an on-premise system or other cloud environment that is remote from the IBM App Connect on Cloud instance. Before installing the client you should ensure that the requirements needed to run it are met. These are listed here https://cloud.ibm.com/docs/services/SecureGateway?topic=securegateway-client-requirements.
Note: The Secure Gateway client uses outbound ports 443 and port 9000 to connect to the npm registry and the IBM Cloud™ environment. Accordingly, make sure that the firewall is open for both ports on the system where the Secure Gateway client is to be installed.
To install and configure the client perform the following steps:
- Install the Secure Gateway client on the required system. Follow the installation instructions that are given at https://cloud.ibm.com/docs/services/SecureGateway?topic=securegateway-client-install
- Add the gateway ID and Security token that were received from the IBM Cloud Administrator to the Secure Gateway client configuration file.
- Check the network connectivity between the system that the SFTP server resides on and the system where the Secure Gateway Client is installed.
- Add an entry of the SFTP server host and port in the Access Control list (ACL) of the Secure Gateway client.
- Start the Secure Gateway client.
- Check the logs for any errors that may have occurred. Resolve that problem and then restart the Secure Gateway client.
Additional documentation that will help when configuring and using the Secure Gateway client can be found here:
Once the SFTP server connectivity is tested, the SFTP Policy in IBM App Connect Enterprise on IBM Cloud can be created. In that policy, the host and port values that are specified should be the cloud generated host and port from the secure gateway destination. The rest of the information in the policy will remain the same.
Acknowledgement and thanks to the following for their contribution to this article
Kamal Kishor Choubey