The SSH File Transfer Protocol or Secure File Transfer Protocol (SFTP) is a client-server network protocol that enables the secure access, transfer, and management of files between remote systems. SFTP runs over the Secure Shell (SSH) protocol.

Using IBM App Connect as a client, you can establish a secure communication channel to an SFTP server to perform file-based operations, and can pass key data between the SFTP server and other apps – automatically, in real time. You can do so using configuration and data mapping without a need for coding, and can achieve a return on your investment in minutes/hours, not days/months.

This guide shows you how.

If you can’t find what you want, or have comments about the “how to” information, please either add comments to the bottom of this page or .

A business scenario

The challenge

Let’s suppose you work in the Human Resources (HR) department of a large corporation. Your company advertises job vacancies online, and interested candidates are required to submit their applications with an attached CV by using an online form on your company’s website. Your HR department handles a lot of confidential data – so you’re using an SFTP server for secure file transfer and management. To help process submitted job applications, you’re looking a fast and efficient way to transfer the CV attachments to the SFTP server for processing.

How App Connect can help

Free up your time and eliminate the monotony of manual file transfers by using App Connect to automate the process. You can connect App Connect to your SFTP service, form designer app, and other apps, and let it do the work for you.

So when online applications are submitted, attached CVs get automatically transferred to folders in an SFTP location that pertain to specific job roles. Perhaps you also want to generate instant messages to department heads or team leads to make them aware that new CVs have been uploaded for their attention. And if you’re considering backing up or archiving all actioned CVs to another secure file-sharing service, App Connect can help with that too!

What should I consider first?

Before you use App Connect Designer with SFTP, take note of the following considerations:

  • You can connect to on-premises SFTP servers that are in a private network, or to cloud-hosted SFTP servers.
  • If the SFTP server is in a private network (for example, behind a firewall), you’ll need to set up a gateway that App Connect will use to securely access the SFTP server. You can use the IBM Secure Gateway Client to set up the required network connection for accessing protected data.

    If you’ve previously used the Secure Gateway Client to set up a network connection for an App Connect application that is on the same private network as the SFTP server, you can use this network connection with the SFTP server. If you don’t have such a network connection in place, configure one as described in Configuring a private network for IBM App Connect. Also ensure that the SFTP server’s host and port are defined in the Secure Gateway Client’s access control list.

  • To create an integration flow that passes key data between an SFTP server and other apps, you must connect App Connect to each app in the flow. To connect App Connect to an SFTP server, you’ll need the following connection details. You might need to contact your server administrator for some of these details.
    • SFTP service URL: If connecting to an on-premises SFTP server in a private network (through a secure gateway), specify the base URL with the server host name (or IP address) and the port number of the server listener. If you are using the default SSH port of 22, this value can be omitted. Use the following format for the URL:
      http://hostname:port
      For example:
      http://myserver.example.com
      http://192.0.2.0:22

      If connecting to a cloud-hosted SFTP server, specify the instance host name or custom subdomain (without an http:// or https:// prefix). For example:
      mysubdomain.sftpprovider.com

    • Username: A valid user name for authenticating to the SFTP server. This user must have the relevant permissions required to perform the supported App Connect actions on the folders and files you want to work with.
    • Password: The associated password for the user name.
    • Host public key: Optional. The SFTP server’s public key in MD5 format that carries the identity of the server. (Used to verify that the correct SFTP server is being connected to.)
    • Network name: The name of the network (configured using the IBM Secure Gateway Client) for connecting to an SFTP server in a private network. If you are connecting an on-premises SFTP server, select a network name. You can leave this field blank if connecting to a cloud-hosted SFTP server.

      Important: Ensure that the Secure Gateway Client has been started. If you need to, you can start the Secure Gateway Client as described in Configuring a private network for IBM App Connect: Finally, start and configure the Secure Gateway Client.

    • Example of completed fields for connecting App Connect to an on-premises SFTP server

      Example of completed fields for connecting to SFTP

    • Example of completed fields for connecting App Connect to a cloud-hosted SFTP service

      Example of completed fields for connecting to SFTP

    You can connect to the SFTP app either from the Applications tab on the App Connect Catalog page, or when you add the app to a flow.

    Tip:

    • Immediately after you connect, rename your account with a meaningful name that helps you identify which server you are connected to. You can rename an account only from the Applications tab on the Catalog page, and before it’s used in a flow.
    • If you’d like to access multiple SFTP servers in your flows, you’ll need to add an account for each server.

      For information about renaming accounts and setting up multiple accounts, see Managing accounts in App Connect.

  • To use SFTP as the source application in a flow, you’ll need to configure the SFTP polled event that should trigger the flow. You can poll the SFTP server at specific time intervals for new or updated folder items, and then configure a target application to perform an action whenever such files or folders are detected. The polling mechanism queries time stamps to detect changes to your folder items, and uses the time stamps that are generated when a file or folder is created or updated in the SFTP server. It’s therefore important to ensure that the time zone configured for the SFTP server matches the time zone specified in App Connect when configuring polling. For example, if you live in a country where the clocks change (for example, for Daylight Saving Time), you might need to reset your time zone to accommodate the offset at the relevant time of year. For more information about configuring polled events, see Configuring polled events to trigger flows.

    Note: For the ‘New or updated Folder item’ SFTP event, the polling mechanism will query the following directories for new or updated folder items:

    • If you logged in to the SFTP server (using your App Connect account) as an admin or root user, the directory polled will be the root folder.
    • If you logged in to the SFTP server (using your App Connect account) as an non-admin user, the directory polled will be the /user/your_username location.
  • To use SFTP as a target application, you can choose from a set of pre-configured actions for files, folders, and folder items.

    Specifying paths

    For all file-based operations, you’ll need to specify a location for the file or folder you’re working with, and will be required to specify one or more of these path types:

    • Source Path: Identifies the location where you want to create a file or folder, the location of a file whose contents you want to retrieve, or the location of a file or folder you want to rename or delete.
    • Destination Path: Identifies the location to which you want to save a renamed file or folder.
    • Parent Path: Identifies the location of a folder whose ‘folder items’ you want to retrieve. You’ll obtain a listing of the child folders and files directly under the specified parent location together with their metadata.

    Paths are specified as UNIX-style paths, and can be added as hard-coded paths or mapped values.

    • For hard-coded paths:
      • If specifying the source or destination path of a file, use the the absolute path from the root directory, including the file name (where the root directory denotes the topmost directory in the tree hierarchy).
      • If specifying the source or destination path of a folder, use the the absolute path from the root directory, including the folder name.
      • If specifying the parent path, use the the absolute path from the root directory, including the folder name for which you want a listing of child items.
    • For mapped paths, you can use the Insert a reference Insert a reference icon icon to map to a value from a previous node in your flow.

      If mapping to output from a previous SFTP node in a flow, you can map to paths or filenames that were returned by the SFTP event or action. For example:

      Mapped field Meaning
      SFTP Path mapping or

      SFTP Source Path mapping

      Depending on the earlier SFTP event or action, a Path or Source Path mapping could denote the absolute path from the root directory, including a file name (for example, of a created or downloaded/retrieved file). Or, it could denote the absolute path from the root directory, including a folder name (for example, of a created folder).
      SFTP Parent Path mapping This mapping denotes the absolute path from the root directory, of a parent location. For example, for the /somewhere/someplace/newfile.txt file, this mapping would resolve to /somewhere/someplace.
      SFTP Name mapping This mapping denotes a file or folder name.
      SFTP File Name mapping This mapping denotes a file name.

    • Examples for specifying hard-coded paths

      The following examples are based on the directory structure shown in this image.

      Example SFTP server directory structure

      Action Field selections
      Create a file named myfile2.txt in the root location. Source Path: /myfile2.txt
      Retrieve the contents of (or download) the file named myfile.txt from the root location. Source Path: /myfile.txt
      Rename the file named filename.txt file in the somewhere location to filename_UPDATED.txt. Source Path: /somewhere/filename.txt
      Destination Path: /somewhere/filename_UPDATED.txt
      Create a folder named sometime in the someplace location. Source Path: /somewhere/someplace/sometime
      Rename the folder named in_time within the someplace location to InTime. Source Path: /somewhere/someplace/in_time
      Destination Path: /somewhere/someplace/InTime
      Retrieve a listing (with metadata) of the child files and subfolders directly under the somewhere parent location. Parent Path: /somewhere

      Based on the directory structure in the image above, this should return an array of objects for the /somewhere/someplace and /somewhere/filename.txt folder items. Note that the /somewhere/someplace/in_time folder is omitted because it is not at the top level of /somewhere. Example array returned:

      
      [
      {"path":"/somewhere/someplace",
      "parentPath":"/somewhere","name":"someplace","type":"folder","modifyTime":1521313310000,"accessTime":1521313310000,"userPermission":"rwx","groupPermission":"wx","otherPermission":"wx"},
      {"path":"/somewhere/filename.txt",
      "parentPath":"/somewhere","name":"filename.txt","type":"file","size":131,"modifyTime":1521313287000,"accessTime":1521313287000,"userPermission":"rw","groupPermission":"w","otherPermission":"w"},
      ]
      

    • Examples for specifying mapped paths

      Specify the source path for an SFTP ‘Download file’ action by mapping to a value from an SFTP ‘New or updated Folder item’ event in your flow. Here, the Source Path mapping will resolve to the absolute path from the root directory, including the file name of a newly uploaded file whose contents you want to retrieve; for example, /myfile.txt.

      SFTP Source path field mapping

      Specify the source and destination paths for an SFTP ‘Rename folder’ action by mapping to values from a previous SFTP node in your flow. If the Source Path mapping resolves to /somewhere/someplace, the Destination Path mapping will resolve to /somewhere/someplaceLatest.

      SFTP mapped Path fields

      Specify the source and destination paths for an SFTP ‘Rename file’ action by mapping to values from a previous SFTP node in your flow. If the Source Path mapping resolves to /myfile.txt, the Destination Path mapping will resolve to /PROCESSED_myfile.txt.

      SFTP mapped Path fields

      Specify the source path for an SFTP ‘Delete file’ or ‘Delete folder’ action by mapping to a value from a previous SFTP node in your flow. The file identified by the mapped path, or the identified folder and its contents will be deleted.

      SFTP mapping for a 'Delete with Where' operation

Specifying file types

  • Type of file: Defines the format of a file that you want to create, or whose contents you want to retrieve, as text or binary.

Specifying file contents

  • File Data: Specifies the data to be written/stored to a new file. To prevent corrupted content, ensure that this data matches the specified file type (text or binary).
    • For a text file type, use the File Data field to enter plain text or a mapped value from a previous node.
    • For a binary file type, use the File Data field to specify a mapped value from a previous node.

Note: The ‘Download file’ action enables you to retrieve the contents of a file in a folder. The maximum permitted file size for content retrieval is 10 MB.

Example

SFTP, CSV parser, and IBM Db2 flow image

Create an event driven flow that creates records in an IBM Db2 database by using the parsed contents of a new CSV file in an SFTP server

Learn how to use App Connect Designer to create an event-driven flow that parses the contents of a new comma-separated values (CSV) file in an SFTP server, creates records in an IBM Db2 database from the parsed contents, and then renames the file to indicate it’s been processed.

Join The Discussion

Your email address will not be published. Required fields are marked *