Learn how to use IBM App Connect on IBM Cloud to create an event-driven flow such that on a weekly schedule files are copied automatically from a Dropbox folder to a new bucket in IBM Cloud Object Storage.

IBM’s Cloud Object Storage (COS) S3 is ideal for holding large amounts of colder production data, such as backups and archives, and very large individual files, such as video files, image files, and genomic data. IBM’s Cloud Object Storage S3 is a reliable, durable, and resilient object storage.

Note: This document assumes that you have signed up for free or trial accounts for accounts with Dropbox and IBM Cloud Object Storage, or that you have business accounts.

Additional reference:

Scenario:

You use Dropbox to share files in folders specific to users. Once a week, on a defined schedule, you want to copy the files from a shared folder in Dropbox to a new bucket IBM Cloud Object Storage S3. The new bucket is given a name based on the date that the files were copied.

(Click image to view full size.)

Time required:
If you have already connected Dropbox and IBM Cloud Object Storage S3 to App Connect, the steps to create and test the integration flow in this tutorial should take only a few minutes.

First, find or create everything you need:

  1. An App Connect on IBM Cloud service. You can use an App Connect service that you already have or can sign up for App Connect.

    In this example, we use a new free App Connect – Lite plan service.

  2. Obtain account details for connecting App Connect to your applications (If you haven’t already connected App Connect to your applications.):
    • You can create a connection to Dropbox either using a Dropbox user ID and password or using a Google account.
    • To connect to an IBM Cloud Object Storage account, you need the following values that can get from the “Endpoint” and “Service Credentials” pages in the side navigation of your IBM Cloud Object Storage dashboard:
      Endpoint URL        COS Endpoint / PUBLIC or PRIVATE URL value for your region; for example, s3.us.cloud-object-storage.appdomain.cloud (PUBLIC for us-geo region)
      Secret Access Key   COS Service credentials / View credential (with HMAC) / cos_hmac_keys/ secret_access_key
      Access Key ID       COS Service credentials / View credential (with HMAC) / cos_hmac_keys / access_key_id
      Region              COS Endpoint / Select Location value; for example, us-geo
      

      Note: To get a Secret Access key and Access Key ID, you need to have a credential created with the option to generate HMAC credentials. (When adding a new credential, specify {"HMAC":true} in the field “Add Inline Configuration Parameters (Optional):”. For more information, see doc link after examples).

      For example:

      Credential with HMAC keys on IBM Cloud Object Storage console. (Click image to view full size.)

      In App Connect, when creating a connection to IBM Cloud Object Storage:

      Endpoint URL        s3.us.cloud-object-storage.appdomain.cloud
      Secret Access Key   12e3f4b56b780503172f70fdb2d1605a5dcfaf834d896c42
      Access Key ID       1234eeaa5ce67e2a89f52bc33f6a172d
      Region              us-geo
      

      For information about determining these values and about generating HMAC credentials, see “Service credentials” in IBM Cloud Docs / Cloud Object Storage.

Next, create your flow:

  1. Log in to App Connect Designer; for example, launch App Connect through your IBM Cloud dashboard.
  2. Click the Catalog tab, and there connect App Connect to Dropbox and IBM Cloud Object Storage. This enables you to concentrate on configuring the event and actions in the flow editor, using the connection accounts you’ve already created. (You can connect applications in the flow editor, but you should find it more convenient to use the Catalog tab to create and manage connection accounts.)

    The following steps assume that you have used the Catalog tab to connect App Connect to Eventbrite, Sharepoint, and Slack.

  3. From the Dashboard, click New > Event-driven flow.

    Complete the following steps. As you progress, App Connect automatically saves your changes. If you navigate away from the flow at any stage, the flow is saved as a draft flow that you can complete at another time.

  4. Enter a name that identifies the purpose of your flow; for example: Weekly copy Dropbox files to new COS S3 bucket named with date
  5. Configure a scheduler as the event that is to trigger the flow once a week.

    In this example, we want the flow to run every week, on Friday at 8pm GMT. For testing, we also want the flow to run when its first started.

    1. Click the Toolbox tab
    2. Click Scheduler
    3. Click ‘Select schedule type’ and then select the Calendar option
    4. Configure the ‘Run every’ options as Week on FRI at 20
    5. For ‘Timezone’, select Etc/GMT
    6. Select the ‘Also run the flow when it’s first switched on’ checkbox

    (Click image to view full size.)

  6. Click the (+) and then scroll down to Dropbox.
  7. Select Folders > Retrieve all folders as the action.

    Actions on folders/files in Dropbox are by ID, but most people find it easiest to think of file names. The ‘Retrieve all folders’ action returns details of all folders (as a JSON object) that allows App Connect to map from a folder name to its ID.

    If your Dropbox account has more than 10 folders, configure the ‘Maximum number of items to retrieve’ option. You can also change the options for if the maximum number of items (folders) is exceeded, or if no items are found.

  8. Configure a variable to handle the ID of the one folder, with “path_lower”=”/shared”, that we want to process.
    Hint: You don’t need to use a variable, but this example shows how a variable with a meaningful name can be useful instead of using a property reference and expression in multiple places.
    1. Click the (+) and then click the Toolbox tab.
    2. Select ‘Set variable’.
    3. Click Add property, and then give it the name FolderID.
    4. Click Edit mappings, and then map the ID field from the Dropbox / Retrieve folders action. For example, type id and then select ID from under foldercollection.

      The Retrieve folders action returns an array of data for one or more folders (as indicated by the message under the FolderID field), so use a JSONata function to get the ID of the folder with the “path_lower”=”/shared”.

    5. In the FolderID field, click the ID mapping, and from the menu offered click Edit expression. This shows the JSONata expression used to get ID values for all folders as an array:
      {{$DropboxRetrieveallfolders.id}}

      To restrict the data to the folder with “path_lower”=”/shared”, change the expression to the following:

      {{$DropboxRetrieveallfolders[path_lower="/shared"].id}}

    (Click image to view full size.)

  9. Click the (+) and then scroll down to Dropbox, to add the action to retrieve all the files in the folder to be processed.
  10. Select Files > Retrieve all files as the action.
  11. Filter the action to only get files from the folder with the ID determined earlier:
    1. Click Add condition
    2. Click in the field after ‘equals’, type folderif, pause briefly, and then from the menu offered select the variable FolderID.
  12. Click the (+) and then scroll down to IBM Cloud Object Storage S3.
  13. Select IBM Cloud Object Storage S3 > Create bucket as the action, to create a bucket based on the date that the action is performed.

    Configure the action as follows:

    1. Click in the Bucket name field.
    2. Click the Insert a function icon.
    3. From under ‘Date/time functions’, select $now().
    4. (Optionally) configure the $now() function and add text or add field mappings for the name you want to use for the COS S3 bucket. In this example, we added text and edited the $now() function to change the format of the date in the bucket name:
      dearliza-{{$now('[Y0001]-[M01]-[D01]') }}

      For more information about the $now() function, see the JSONata documentation.

    Create bucket action in the flow, showing $now function inserted. (Click image to view full size.)

  14. Add a For each loop to process each file from the Dropbox / Retrieve all files action
    1. Click the (+) and then click the Toolbox tab.
    2. Select ‘For each’.
    3. Click in the ‘Select the collection of items to process: ‘ field
    4. Type filecoll, pause briefly, and then from the menu offered click filecollection from under the Retrieve files action.

      Leave the options to default, to process all items (files) in the collection sequentially and to continue processing other files if an error occurs while processing a file.

      (Click image to view full size.)

    5. In the For each node, configure what should happen for each Dropbox file.

      In this example, we added an If node to take different action depending on the suffix of the Dropbox file (because Dropbox does not provide different Content- Type values for different types of files):

      • If filename contains .mp3 or .pdf then handle this file as a binary file
        1. Dropbox / Download file content – Get the content of this file, by its file path and as file type=binary
        2. IBM COS S3 / Create object – Create an object in the selected bucket, with the file name of this file, and the content from Dropbox.
      • Else (not mp3 or pdf) handle the file as a text file
        1. Dropbox / Download file content – Get the content of this file, by its file path and as file type=text
        2. IBM COS S3 / Create object – Create an object in the selected bucket, with the file name of this file, and the content from Dropbox, setting the content-type=text/plain.

      For each node in the flow, showing the If node with alternative actions for different file types. (Click image to view full size.)

      1. In the For each node, click the (+), click the Toolbox tab, and then click ‘If (conditional)’
      2. In the ‘if’ field, type name, pause briefly, and then select Name from under filecollection
      3. Click Name, then select ‘Apply a function’
      4. From under String functions, select $contains
      5. In the $contains expression, change the missing ‘chars’ with .mp3. This completes the expression as follows:
        {{$contains($Foreachitem.name, ".mp3")}}

        (You can copy and paste this into the ‘if’ field.)

      6. Change the ‘equals’ condition to an ‘is true’ condition
      7. Add another condition for .pdf files. Click ‘Add condition, and then use the same steps as for .mp3 files; for example, to add the expression:
        {{$contains($Foreachitem.name, ".pdf")}}
      8. Next to the ‘if’ label, change the condition to ‘Any of the following are true’
  15. Configure the actions for the If branch of the For each node:
    1. Click the (+) in the If branch of the For each node
    2. Scroll down to Dropbox and then select Files / Download file content
    3. Click in the Path field, type path, pause briefly, and then select ‘Path lower’ from under ‘For each … filecollection’
    4. For the File Type field, select Binary
    5. Click the next (+) in the If branch of the For each node
    6. Scroll down to IBM Cloud Object Storage S3 and then select Objects / Create object.
    7. Configure the Create object action:
      • Bucket: Normally, you’d select an existing bucket from the list offered, but for this flow we want to specify a bucket that does not yet exist. To do this, we use advanced mode.
        1. Click the Map options menu (⋮) and then select Switch to advanced mode.

          (Click image to view full size.)

        2. Click in the Bucket field and enter the same name as you used on the Create bucket action (you can copy the value from the Bucket name field of that action); for example:
          dearliza-{{$now('[Y0001]-[M01]-[D01]') }}
      • Object name: Click in the field and then insert a reference to the ‘For each / filecollection / Name’ value.
      • Object content: Click in the field and then insert a reference to the ‘Dropbox / Download file content / File content’ value.
      • Content type: You can leave this empty.
  16. Configure the actions for the Else branch of the For each node:
    1. Click the (+) in the If branch of the For each node
    2. Scroll down to Dropbox and then select Download file content
    3. Click in the Path field, type path, pause briefly, and then select ‘Path lower’ from under ‘For each … filecollection’
    4. For the File Type field, select Text
    5. Click the next (+) in the If branch of the For each node
    6. Scroll down to IBM Cloud Object Storage S3 and then select Objects / Create object.
    7. Configure the Create object action:
      • Bucket: Normally, you’d select an existing bucket from the list offered, but for this flow we want to specify a bucket that does not yet exist. To do this, we use advanced mode.
        1. Click the Map options menu (⋮) and then select Switch to advanced mode.
        2. Click in the Bucket field and enter the same name as you used on the Create bucket action (you can copy the value from the Bucket name field of that action); for example:
          dearliza-{{$now('[Y0001]-[M01]-[D01]') }}
      • Object name: Click in the field and then insert a reference to the ‘For each / filecollection / Name’ value.
      • Object content: Click in the field and then insert a reference to the ‘Dropbox / Download file content 2 / File content’ value. (Make sure you select ‘File content’ for the Download file content action on the Else branch.)
      • Content type: Enter text/plain

Finally,test your flow:

  1. From the banner, open the options menu [⋮] and click Start flow. Then click Dashboard to exit the flow. Check the flow tile on your Dashboard for indication that your flow has run successfully (the flow tile shows a green tick after a short time).

    Flow tile showing last successful run indicator and message (Click image to view full size.)

    If your flow fails to run successfully, check the log for messages and take appropriate action.

  2. Log in to your IBM Cloud Object S3 instance, and check for the new bucket created and in that the files copied from Dropbox.

    IBM Cloud Object Storage S3, showing the new bucket created by the last successful run of the flow (Click image to view full size.)

Join The Discussion

Your email address will not be published. Required fields are marked *