Learn how to use IBM® App Connect Enterprise on IBM Cloud™ to write or transfer files in the directories available on an SFTP Server over the public internet. This SFTP server is accessible using the username, SSH Identity file and passphrase option.
Your company wants to process business data and store the result in a file that is located in a remote location that is exposed using the SSH File Transfer Protocol (SFTP). This protocol is used as the reading and writing of file data is taking place over the public internet.
An integration solution needs to be constructed that securely writes data to the file on the SFTP server when the processing has taken place.
In this tutorial an existing integration solution, that presents a REST API, is imported to run in IBM App Connect Enterprise on IBM Cloud to do such processing. The file transport used is SFTP, which is secure as it runs on top of the SSH protocol. This is important as the communication between the integration running in IBM App Connect Enterprise on IBM Cloud and the SFTP server is across the public internet and should not be in plain text. Authentication with the SFTP server is done using a username, SSH identity file and passphrase option.
In those cases where the SFTP server is not accessible on the public internet then a secure gateway facility can be used to provide a secure access between the different networks. This is discussed at the end of this tutorial.
This scenario presents one type of usage but it can easily be extended to meet your own needs.
First, find or create everything you need:
- An SFTP Server available over the public internet that has remote directories.
You’ll need the following information, that will be provided by the SFTP Server Administrator:
- Host name of the server
- Port used by server for TCP/IP communication; for example, 22
- Directory location
- SFTP Server Username
- SFTP Server SSH Identity File
- Passphrase for SSH Identity File
- An IBM Cloud account with an IBM App Connect service that provides enterprise integration capabilities.
- Optionally, you can also install IBM App Connect Enterprise for Developers on your local development environment if you want to look at and change the deployed message flow. It is not necessary for running the tutorial, but it is useful for examining the message flow to understand how it works.
(Note: The integration project for this tutorial was developed with the IBM App Connect Enterprise Toolkit).
- If you want a bit more information before you start, you can read more about the IBM App Connect Enterprise for Developers edition on the following page: Get started with IBM App Connect Enterprise
Import the integration project into the IBM App Connect Enterprise Toolkit, to look at the message flow:
You do not need to complete this step unless you want to look at or change the integration service. A separate BAR file is also provided for deploying the integration into IBM App Connect Enterprise on IBM Cloud, as described in a later step.
All the integration service resources required for this tutorial are provided in a project interchange file named CustomerInventory-SFTPwithKey.zip
The steps to import the integration service into the IBM App Connect Enterprise Toolkit are given below:
- Download the project interchange file by clicking the link above and saving the file to a local directory.
- Open the IBM App Connect Enterprise Toolkit.
- Import the project into the toolkit by clicking File > Import. Expand IBM Integration, select Project Interchange, and then click Next.
- Browse to select the downloaded project interchange file, and then click Finish.
- A new project called CustomerInventory, which implements a REST API, is displayed in the IBM App Connect Enterprise Toolkit:
You can view the REST service definition by double-clicking REST API Description.
The REST service has one subflow called getCustomers.subflow that implements a “GET customers” operation. It retrieves selected customer details from an in-memory data structure. The message flow performs the following actions:
Get Customersnode builds a data structure with the requested customer information in it. This is then passed to the
Write to InventoryFileOutputNode which writes the customer information to a file within a directory that is located on the SFTP Server. If the file write is successful, then the same customer information is sent back to the user that invoked the REST API as a success response.
The directory name that is configured on the Basic tab of the FileOutput node must start with a prefix of /tmp. This is a requirement that is specific to IBM App Connect Enterprise on IBM Cloud and it is because of the way the feature is implemented, and locations available, when running in the managed service.
You will also need to specify the same directory structure later when you will create an SFTP Policy in the IBM App Connect Enterprise on IBM Cloud instance.
In the same policy you will also be required to provide the same Security identity name that is specified in the Security identity box in the FTP Tab for the FileOutput node. For example, DummyIdentitywithkey in this case.
The following images show the properties on the Basic and FTP tabs for the the
Write to InventoryFileOutputNode
Write to Inventory Basic Tab:
Write to Inventory FTP Tab:
Configure the integration in IBM App Connect Enterprise on IBM Cloud:
- Download the zip file CustomerInventory-SFTPwithKey.zip and extract the BAR file named CustomerInventory.bar from it.
- If necessary, sign in to IBM Cloud. From the IBM Cloud dashboard, select and then launch your App Connect service instance.
- From the App Connect on IBM Cloud dashboard, click New > Import a BAR file and select the BAR file that you extracted. Then click Import.
The integration server that is created as a result of the import is displayed in the dashboard.
It is now necessary to create an SFTP policy so that property overrides can be specified.
- From the App Connect menu, click , then Manage > Policies, to open the Policies view.
- Click New policy.
- Create an SFTP(Security Identity) policy with the following details:
- Policy name: Enter
DummyPolicywithKey. You can change it to any name based on naming conventions and standards.
- Security Identity: This should be same value that is specified in the Security identity box in the FTP Tab of the FileOutputNode. For example in this case it is
DummyIdentitywithkeyto match with the value shown in the screen shots of the node properties, that were shown in the previous section.
- User name: This is the SFTP Server user-name that the message flow will use to identify with the SFTP server
- Use identity file: Enable this checkbox to upload the SSH identity file. Once this checkbox is enabled, two drop boxes will become visible, one for ‘Identity file’ and one for ‘Passphrase’ Under the ‘Identity file’ box, click on ‘Add a file’ to select and then upload the identity file to be used by this policy. This SSH identity file is used for authentication against the SFTP server.
- Passphrase: This is the passphrase to be used for SSH identity file while connecting to SFTP Server and a value must be supplied.
- Directories to create: This is the list of the names of directories that need to be created as part of the configuration when this message flow is run. All directories must have a prefix of /tmp and this will be automatically added to the entered text as soon as the ‘Add’ icon is clicked. In this case the directory name entered needs to match the value supplied for the Directory value on the Basic tab of the
Write to InventoryFileOutputNode.
- Policy name: Enter
- Click Create and then return to the App Connect Enterprise on IBM Cloud dashboard.
- To attach the policy to the uploaded BAR file, search for the policy name in the search bar and select ‘Attach policy’ from the actions.
- In the pop-up window check the ‘CustomerInventory’ Integration Server and click Apply.
- Once selected, the value for ‘Used by’ will change from ‘0 integration servers to ‘1 integration server.
- Return to the dashboard. Then start the integration by opening the options menu [⋮] for the integration server, and then clicking Start. When the integration shows Running, the integration is running and ready to use.
Finally, test your integration:
To test the integration you will run a GET operation on the getCustomers REST API to retrieve data from the in-memory table. This data is then formatted as an XML message and written to the file on the SFTP and also returned to the user.
- From the dashboard, click on the integration server tile to open the integration.
- Click Show API Explorer.
- Only one possible API is shown. Click on that (Get /Customers) followed by clicking on ‘Try It’ tab.
- Click Send and success should be returned. The result should look like this:
This indicates that the whole message flow ran successfully. To verify this it is necessary to login to the SFTP server and see if the file is present in the expected directory.
Use a tool such as WinSCP as an SFTP Client to login to SFTP Server.
Start the tool, specify the SFTP server host name and the username to authenticate to the SFTP server with.
Click on Advanced -> SSH -> Authentication – In the Private Key File, click […] and select the path to the key file.
Click Ok followed by clicking on Login and provide the passphrase for the SSH key file.
After successfully logging in, navigate to the directory location.
Open the file to check the contents of the file.
If all of these steps work, then you have the tutorial up and running successfully. If it fails, then an error message with the reason will be returned. Check the integration log for any errors – you can access the logs from the ’Manage > Logs’ menu item in the dashboard.
If you want to extend this tutorial then you could change the message flow to retrieve data from an on-premise or cloud hosted Oracle database instead of using the in-memory table that is currently used. To make this change you are recommended to see the tutorial Using IBM App Connect Enterprise on IBM Cloud to retrieve data from an Oracle Database
Connecting to SFTP Server using IBM Secure Gateway:
If the SFTP Server is hosted on-premises or in a private cloud (i.e. not accessible using the public internet), and you do not want to open a separate port on the firewall, then you need a service that is able to provide secure access between the different networks. IBM Secure Gateway Service is such a service. It provides connectivity to on-premise private networks or environments available in private cloud and it can be used to securely access the SFTP Server. You can read more about the Secure Gateway on the following page: Getting started with IBM Secure Gateway
This diagram shows the configuration that you would use when using IBM Secure Gateway Service
From the IBM Cloud, complete the following step to configure the Secure Gate service to connect to an SFTP Server over the public internet:
- Create one gateway in an IBM Secure Gateway service instance provisioned in IBM Cloud.
- Ask the SFTP Administrator to install the Secure Gateway Client in the SFTP server environment. See the section Secure Gateway Client Installation and Configuration below to see more details on this task.
- Share the gateway id and Security token with SFTP Administrator to configure it in the IBM Secure Gateway client. Once configured, the gateway will be connected to client using the provided gateway ID.
- Ask the SFTP Administrator to provide the SFTP Server host name/IP and SFTP server port detail, that will be configured in the destination.
- Create a destination in the IBM Secure Gateway service instance with the SFTP server host and SFTP server port provided by SFTP Administrator.
- In the case that an ‘Access is blocked’ sign appears on the destination ask the SFTP Administrator to add the SFTP server host and SFTP server port entry in the ACL to allow the communication.
The next step is to configure the Secure Gateway client
Secure Gateway Client Installation and Configuration
The Secure Gateway client needs to be installed on a system that is adjacent to the SFTP server. This would most likely be on an on-premise system or other cloud environment that is remote from the IBM App Connect on Cloud instance. Before installing the client you should ensure that the requirements needed to run it are met. These are listed here https://cloud.ibm.com/docs/services/SecureGateway?topic=securegateway-client-requirements.
Note: The Secure Gateway client uses outbound ports 443 and port 9000 to connect to the npm registry and the IBM Cloud™ environment. Accordingly, make sure that the fire-wall is open for both ports on the system where the Secure Gateway client is to be installed.
To install and configure the client perform the following steps:
- Install the Secure Gateway client on the required system. Follow the installation instructions that are given at https://cloud.ibm.com/docs/services/SecureGateway?topic=securegateway-client-install
- Add the gateway ID and Security token that were received from the IBM Cloud Administrator to the Secure Gateway client configuration file.
- Check the network connectivity between the system that the SFTP server resides on and the system where the Secure Gateway Client is installed.
- Add an entry of the SFTP server host and port in the Access Control list (ACL) of the Secure Gateway client.
- Start the Secure Gateway client.
- Check the logs for any errors that may have occurred. Resolve that problem and then restart the Secure Gateway client.
Additional documentation that will help when configuring and using the Secure Gateway client can be found here:
Once the SFTP server connectivity is tested, the SFTP Policy in IBM App Connect Enterprise on IBM Cloud can be created. In that policy, the host and port values that are specified should be the cloud generated host and port from the secure gateway destination. The rest of the information in the policy will remain the same.
Acknowledgement and thanks to the following for their contribution to this article
Kamal Kishor Choubey