Kubernetes with OpenShift World Tour: Get hands-on experience and build applications fast! Find a workshop!

Leverage deep learning in IBM Cloud Functions

Based on Apache OpenWhisk, IBM Cloud Functions is a Functions as a Service (FaaS) platform that makes it easy to build and deploy serverless applications.

In this tutorial, you’ll build a serverless application using IBM Cloud Functions that monitors the content of a Cloud Object Storage bucket and analyzes the content of images that are uploaded to the bucket by a human or an automated process. For illustrative purposes, analysis is performed by a deep learning microservice from the Model Asset eXchange and analysis results are stored as JSON files in the same bucket.

tutorial scenario

You can easily adapt the outlined approach to take advantage of hosted cognitive services, such as those provided by IBM Watson®, and to store results in a NoSQL datastore like Cloudant® or a relational database.

Learning objectives

By completing this introductory tutorial, you learn how to monitor a Cloud Object Storage bucket for changes (new objects, updated objects, or deleted objects) using Cloud Functions and how to use deep learning microservices from the Model Asset eXchange to automatically analyze those objects in near real time.

Upon completion of the tutorial, you know how to use the IBM Cloud CLI to set up change monitoring for a Cloud Object Storage bucket and how to derive information from uploaded objects, such as images, audio, video or text files in near real-time using deep learning microservices.

Prerequisites

To follow this tutorial, you must have:

  • An IBM Cloud account.
  • Administrator access to a Cloud Object Storage instance. (You can follow this tutorial using Lite or Standard plan instances.) The tutorial instructions refer to an instance named cloud-object-storage-lite. Whenever necessary, replace the the name with your Cloud Object Storage instance name.
  • An installation of the IBM Cloud CLI version 0.19 or above. Refer to these instructions on how to check your installed version and upgrade.
  • An installation of the IBM Cloud Functions plug-in version 1.0.34 or above.

If you have never used Cloud Functions, briefly review the concepts before proceeding.

Download the tutorial artifacts from GitHub

If you have a GitHub client installed on your local machine, clone the tutorial repository:

git clone https://github.com/IBM/use-deep-learning-in-ibm-cloud-functions
cd use-deep-learning-in-ibm-cloud-functions

If you don’t have a GitHub client installed, download the ZIP file and extract it.

Estimated time

Completing this tutorial should take about 30 minutes.

Steps

To implement the serverless application, you need to create three Cloud Function entities:

  • A Cloud Object Storage trigger that fires whenever an image is uploaded to a Cloud Object Storage bucket
  • An action that analyzes the image and stores the analysis results in the same bucket
  • A rule that associates the trigger with the action

trigger rule action schematic

First, create a bucket in your Cloud Object Storage instance.

Create a regional bucket

The serverless application you are about to create utilizes Cloud Object Storage triggers. These triggers support regional buckets in the us-east, us-south, or eu-gb region but not cross-region buckets, single-site buckets or regional buckets in other regions.

  1. Open your IBM Cloud resource list in your web browser and locate your storage services.
  2. Select your Cloud Object Storage instance cloud-object-storage-lite.
  3. Create service credentials (assigning writer access) that the serverless application will use to access the bucket. Create service credentials

  4. Create a regional bucket in us-east, us-south, or eu-gb.

    Create a regional bucket

Throughout this tutorial, the instructions refer to a bucket named vacation-pictures. Buckets in the public cloud share a global namespace, meaning no two users can create a bucket with the same name. Therefore you must choose a different bucket name and use your bucket name when you follow the tutorial steps.

Configure your IBM Cloud CLI

  1. Open a terminal window, launch the IBM Cloud CLI, and log in using your IBM ID:

    ibmcloud login

  2. Display the region, resource group, and Cloud Foundry API endpoint that your IBM Cloud CLI is targeting.

    ibmcloud target

    The output might look as follows:

    API endpoint:      https://cloud.ibm.com
    Region:            us-south
    User:              somebody@somewhere.com
    Account:           Somebody's Account (7...g)
    Resource group:    No resource group targeted, use 'ibmcloud target -g RESOURCE_GROUP'
    CF API endpoint:
    Org:
    Space:
    
  3. If the listed region doesn’t match the region where you’ve created the bucket run the following command, replacing us-south with your region:

    ibmcloud target -r us-south

  4. If no resource group target is set, run

    ibmcloud target -g default

  5. If no Cloud Foundry API endpoint is set, run

    ibmcloud target --cf-api api.ng.bluemix.net

You’re ready to start building the serverless application.

Create a Cloud Object Storage trigger

A Cloud Object Storage trigger is a pre-installed Cloud Functions package. You can use it to listen for changes to objects (add, change, remove) in a bucket. Each time such a change occurs the trigger fires.

Before you can create this trigger, you must create an Identity and Access (IAM) managed namespace and assign the Notifications Manager role for your Cloud Object Storage instance to this namespace.

IAM-managed namespaces are used to group related Cloud Function entities, such as triggers and actions. For example, you might store development entities in one namespace, QA entities in a second namespace, and production entities in a third.

If you have used Cloud Functions before to implement serverless applications, you are probably familiar with Cloud Foundry-based namespaces, which serve the same purpose but don’t support the features you need to implement Cloud Object Storage bucket monitoring.

  1. Create IAM-managed Cloud Functions namespace analyze_images, which will contain the tutorial artifacts. If you choose a different name make sure to specify that name in the commands listed below:

    ibmcloud fn namespace create analyze_images --description "identify objects in images"

    To list your namespaces run ibmcloud fn namespace list.

  2. Set your IBM Cloud CLI context to the analyze_images namespace. Once set, all new Cloud Functions entities will be created in this namespace:

    ibmcloud fn property set --namespace analyze_images

    To list entities in your current namespace run ibmcloud fn list.

  3. Assign the Notifications Manager role to the analyze_images namespace. Once this role is assigned, Cloud Functions can view, modify, and delete notifications for a Cloud Object Storage bucket. The source-service-instance-name parameter identifies the namespace (analyze_images) and target-service-instance-name the Cloud Object Storage service instance (cloud-object-storage-lite) for which the role is granted:

    ibmcloud iam authorization-policy-create functions cloud-object-storage "Notifications Manager" --source-service-instance-name analyze_images --target-service-instance-name cloud-object-storage-lite
    

    If this command fails with an error indicating that the Notifications Manager role was not found, your installed IBM Cloud CLI or Cloud Functions plug-in is too old. Upgrade to the latest version and try again.

Now that the prerequisites are in place, you can create Cloud Object Storage triggers for the selected namespace and Cloud Object Storage instance. Remember that these triggers can only monitor regional bucket changes. Cross-region and single-site buckets are currently not supported.

  1. Create Cloud Object Storage trigger bucket_jpg_write_trigger that fires whenever an image (with the .jpg extension) was successfully added to or modified in Cloud Object Storage bucket vacation-pictures:

    ibmcloud fn trigger create bucket_jpg_write_trigger --feed /whisk.system/cos/changes --param bucket vacation-pictures --param suffix ".jpg" --param event_types write
    

    Let’s review the three parameters:

    • bucket, set to vacation-pictures, identifies the bucket that this trigger will monitor.
    • suffix, an optional parameter (like prefix, which is not used here), is set to .jpg. The parameter is used to define a filter, which causes the trigger only to fire if an object is added or modified whose key ends with .jpg. As a result the trigger will not fire when a video is uploaded, or, as you’ll see later, when we store analysis results in this bucket. NOTE: If you want to filter using multiple suffixes, such as .jpg and .png, you must create one trigger for each suffix (e.g., bucket_jpg_write_trigger and bucket_png_write_trigger). The same applies to prefixes.
    • event_types, another optional parameter, which is set to write, is used to identify the type of change that fires the trigger. If not specified, all changes will fire the trigger.

      In this tutorial, we don’t want this trigger to fire when an image is deleted because no additional insights can be derived from a non-existing picture.

    Refer to the documentation for a complete list of supported trigger parameters and values.

    You can display the list of triggers in your current namespace by running ibmcloud fn trigger list.

  2. Inspect the trigger properties:

    ibmcloud fn trigger get bucket_jpg_write_trigger

Next, you’ll define a Cloud Function action that contains code to process the image.

Create an action

An action is a piece of code (written in one of the supported programming languages, such as Python and Node.js). In this tutorial we’ll create an action that detects objects in an image and stores the results in a JSON file in the Cloud Object Storage bucket where the image is located.

The action requires several input parameters:

  • The name of the image that was added to the monitored bucket or modified in that bucket
  • The bucket name
  • Connectivity information for the Cloud Object Storage instance that the action’s source code can use to download the image from the bucket and to store the analysis results

The image name and the bucket name are provided in the trigger’s payload. However, connectivity information is not. Therefore, you have to create a package, bind your Cloud Object Storage instance to the package (which makes its connectivity information available to actions within the package) and store the analysis action inside the package.

  1. Create a package named manage_pictures:

    ibmcloud fn package create manage_pictures

    You can display the list of packages in your current namespace by running ibmcloud fn package list.

  2. Display the package properties:

    ibmcloud fn package get manage_pictures

    Note that no resource bindings exist:

     ...
     {
      "namespace": "...",
      "name": "manage_pictures",
      "version": "0.0.1",
      "publish": false,
      "binding": {}
     }
    
  3. Bind the cloud-object-storage-lite Cloud Object Storage instance to the manage_pictures package:

    ibmcloud fn service bind cloud-object-storage manage_pictures --instance cloud-object-storage-lite

    The operation will fail if no service credentials are defined for this instance. If multiple credentials are defined you can specify the desired credentials using the --keyname <credentials-name> parameter.

  4. Display the package properties again:

    ibmcloud fn package get manage_pictures

    The output should now indicate that a parameter __bx_creds is defined that actions within this package can access and use to connect to the cloud-object-storage-lite instance you’ve bound to the package in the previous step:

    {
     "namespace": "...",
     "name": "manage_pictures",
     "version": "0.0.1",
     "publish": false,
     "parameters": [
         {
             "key": "__bx_creds",
             "value": {
                 "cloud-object-storage": {
                     "apikey": "...",
                     "cos_hmac_keys": {
                         "access_key_id": "...",
                         "secret_access_key": "..."
                     },
                     "credentials": "...",
                     "endpoints": "https://cos-service.bluemix.net/endpoints",
                     "instance": "cloud-object-storage-lite",
                     ...
                 }
             }
         }],
     ...
    }
    

    You’ll have a chance to review the relevant action source code in a minute, after you’ve created the action.

  5. Create the bucket_write_action in the manage_pictures package using the provided actions/python/detect_objects.py Python source code. Note the kind parameter, which defines the runtime environment Cloud Functions shall use to run the code:

    ibmcloud fn action update manage_pictures/bucket_write_action actions/python/detect_objects.py --kind python:3.7

    You can display the list of actions in your current namespace by running ibmcloud fn action list.

  6. Display the action properties to review the runtime settings:

    ibmcloud fn action get manage_pictures/bucket_write_action

    Your Cloud Object Storage connectivity information should be listed as a parameter.

  7. Review the action’s detect_objects.py source code in the actions/python/ directory.

    The GitHub repository also contains a Node.js implementation of this action (and other actions mentioned later on) in actions/js/.

    The action is implemented in Python and uses the Cloud Object Storage SDK to read and write objects:

    • The image is loaded from the specified Cloud Object Storage bucket into memory.
    • The image is sent to a cloud-hosted Object Detector microservice, which identifies the types of objects that are depicted in the image.

      For illustrative purposes the tutorial application is pre-configured to utilize an evaluation microservice instance that the Center for Open Source Data and AI Technologies hosts. Evaluation instances must never be used for production purposes. If you plan on regularly using a microservice to analyze your data, you should deploy your own copy, for example on the Red Hat® OpenShift® container platform, as described in “Deploy deep learning models on Red Hat OpenShift.”

    • The results that were returned by the microservice are saved in JSON format in the bucket where the image resides.
    • The action returns the bucket name, the object’s key, the annotation type, and the object’s annotation key.

    Note that the action can process images of any encoding, even though the trigger you’ve defined only fires if JPG-encoded images are uploaded. That’s the motivation why the action is named bucket_write_action instead of bucket_jpg_write_action.

At this point, you have created the bucket_jpg_write_trigger trigger and the manage_pictures/bucket_write_action action. However, the trigger is not yet associated with the action and therefore no processing would be performed when the trigger fires.

Create a rule

A rule associates a trigger with an action or a sequence (which is basically a set of actions). When a trigger is fired, the associated action (or sequence) is executed and the triggers’ payload passed as parameters.

  1. Create a rule bucket_jpg_write_rule that associates bucket_jpg_write_trigger with the manage_pictures/bucket_write_action:

    ibmcloud fn rule create bucket_jpg_write_rule bucket_jpg_write_trigger manage_pictures/bucket_write_action

    You can display the list of rules in your current namespace by running ibmcloud fn rule list.

  2. Inspect the rule properties to confirm that the rule is active and associates the trigger with the action as expected:

    ibmcloud fn rule get bucket_jpg_write_rule

Now that the rule was created the action will be executed every time a new JPG image is uploaded to the bucket.

  1. Review the Cloud Function entities that implement your serverless application:

    ibmcloud fn list

    If the output looks as follows, you are ready to test the application:

    packages
    /.../manage_pictures                  private
    actions
    /.../manage_pictures/bucket_write_action private python:3.7
    triggers
    /.../bucket_jpg_write_trigger         private
    rules
    /.../bucket_jpg_write_rule            private              active
    

Test the serverless application

To test the application using the GUI:

  1. Upload a JPG image to the vacation-pictures bucket, e.g. greenland.jpg. After a few seconds a new object named annotations/greenland.json should appear in the bucket.

    Bucket content after images were uploaded

  2. Upload a PNG image (or any other file with an extension other than .jpg) to the bucket. No new JSON file should appear in the bucket because the bucket_jpg_write_trigger didn’t fire.

To monitor execution of your Cloud Function entities:

  1. Open the Cloud Functions dashboard in your browser.
  2. Select the Monitor tab.
  3. Select the analyze_images namespace from the drop down on the right hand side.

    Monitor cloud function execution

  4. Open the activity log output for the manage_pictures/bucket_write_action by clicking on the activation ID (25...64 in the screen capture above). The log contains any output that might have been generated, along with the response, which might look as follows:

    "response":
       {
         "result":
           {
             "annotation_key":"annotations/greenland.json",
             "annotation_type":"max-object-detector",
             "bucket":"vacation-pictures",
             "key":"greenland.jpg"
           },
         "status":"success",
         "success":true
       }
    

Extending the serverless application

You can easily extend or customize the serverless application by changing the action code to perform a different type of analysis (for any kind of media stored in Cloud Object Storage). In conclusion of this tutorial you will change the action code to generate image captions and extend the application to remove the generated JSON file whenever its associated image is deleted from the bucket.

Perform a different kind of analysis

The bucket_write_action (implemented in actions/python/detect_objects.py) you created earlier utilizes a microservice from the Model Asset eXchange that detects objects in images. If you want to perform a different kind of analysis (e.g., generate an image caption), you can modify the action source code to invoke another cloud service to perform the desired task.

The tutorial repository contains an example action implementation actions/python/generate_image_caption.py for the Image Caption Generator microservice. The main difference between this implementation and the actions/python/detect_objects.py is the service URL that’s invoked and the payload.

  1. Update the bucket_write_action to generate image captions instead:

    ibmcloud fn action update manage_pictures/bucket_write_action actions/python/generate_image_caption.py --kind python:3.7

  2. Upload a JPG image to the manage_pictures bucket and verify that the generated JSON file contains text that describes the picture content.

Automatically remove analysis files

You can easily extend the serverless application to remove the automatically generated JSON files whenever an image is deleted from the bucket:

  • Create a trigger that fires whenever a JPG file is deleted from the bucket
  • Create a delete action that removes the JSON file that’s associated with the image
  • Create a rule that associates the delete trigger with the delete action

Complete the following steps:

  1. Create a delete trigger. Note that the event_types parameter is set to delete:

    ibmcloud fn trigger create bucket_jpg_delete_trigger --feed /whisk.system/cos/changes --param bucket vacation-pictures --param suffix ".jpg" --param event_types delete
    
  2. Create a delete action. The action’s sample source code is located in actions/python/bucket_delete_action.py:

    ibmcloud fn action update manage_pictures/bucket_delete_action actions/python/delete_annotation.py --kind python:3.7

  3. Create a rule that associates the trigger with the action:

    ibmcloud fn rule create bucket_jpg_delete_rule bucket_jpg_delete_trigger manage_pictures/bucket_delete_action

  4. Review the newly created Cloud Function entities.

    ibmcloud fn list

    The output should look as follows:

    packages
    /.../manage_pictures                  private
    actions
    /.../manage_pictures/bucket_delete_action private python:3.7
    /.../manage_pictures/bucket_write_action private python:3.7
    triggers
    /.../bucket_jpg_delete_trigger        private
    /.../bucket_jpg_write_trigger         private
    rules
    /.../bucket_jpg_delete_rule           private              active
    /.../bucket_jpg_write_rule            private              active
    

To test the updated application remove a previously uploaded JPG image from the vacation-pictures bucket. The corresponding JSON file should be automatically deleted.

Support additional object types

In this tutorial you’ve created Cloud Function triggers that only fire when JPG-encoded images are uploaded. To support additional image encodings (e.g. PNG) you’d add additional triggers and rules, as illustrated in the create_serverless_app.* shell and batch scripts in the tutorial GitHub repository.

Summary

In this tutorial, you learned how to create a serverless application that monitors the content of a Cloud Object Storage bucket and utilizes deep lerning microservices from the Model Asset eXchange to analyze the content of images.

To learn more about the Model Asset eXchange, take a look at Learning Path: An introduction to the Model Asset eXchange.

To learn more about IBM Cloud Functions and common usecases, refer to the section “What would I use Cloud Functions for?” in the documentation.

Patrick Titzler