Overview

Streams integrates with other technologies using adapters to popular protocols such as TCP, ODBC, Kafka, JMS, MQTT and HDFS, just to name a few. REST is another established protocol that is gaining popularity because of its use in many cloud-based services. This article describes how to use Streams HTTP adapters to integrate SPL applications to Cloudant and other RESTful, web-based services.

Contents

Prerequisites

Grab and install the latest release of the Inet Toolkit on Github. (direct download) Download the Streams Integration Samples JazzHub project and extract the CloudantGet and CloudantPost samples. (direct download)

Setting up Cloudant

Getting connection information

Create a Cloudant account on cloudant.com, or create a Cloudant service on Bluemix. Note the following information.
  • username
  • password
  • host
The host is [username].cloudant.com If you created the service through Bluemix, the username can be found on the “Service Credentials” tab of the Bluemix service dashboard.

Creating a Cloudant database

Navigate to the Cloudant dashboard by pressing Launch in Bluemix or logging in to cloudant.com. Click “Add New Database” in the top right corner. Follow the prompts to name and create the database named “test”. An empty database will be created. Capture2 To view the contents of the database, click the database name link in the table of the “Databases” tab, and select “All Documents”. Cloudant database view

Using Cloudant’s REST API

Putting documents into Cloudant

In Streams Studio, import the CloudantPost sample application into your Streams Studio workspace and launch it using your Cloudant credentials. The url is https://[username].cloudant.com/test . For more information on Submission Time Values, see this article Cloudant post submission time parameters This Streams application will create a new Cloudant document containing the string “Cloudant” every two seconds. For each document, the console will print “errorMessage: Created”, along with some other JSON metadata. Let the application run for a few seconds and then cancel the Streams job. If the request was not successful, a few things might have happened:
  • The console printed a JSON error response. You successfully connected to Cloudant, but some error occurred after that point. The “reason” field gives some hints about what went wrong.
  • If the console printed a non-JSON error or exception, you have not successfully connected to Cloudant. Double check the url for accuracy.
  • See “Troubleshooting unsuccessful requests”
Upon success, new documents have been added to Cloudant. Head back to the “All Documents” view to see them. Database view with document To view the data inside a document, select the “pencil” icon on the top-right corner of the dark-colored “document box”. Cloudant document view You should see the document that you just created. Note that the CloudantPost sample application’s first operator is a Beacon that activates the HTTPPost and gives it the message body data. To change the frequency or number of requests, use the Beacon’s period or iterations parameters. However, the HTTPPost operator can connect with any stream carrying an rstring parameter named “message”.

Listing all documents in Cloudant

Now that we can create documents, we want to access them from the REST API. Launch CloudantGet with the same values, except change the url to [username].cloudant.com/test/_all_docs . Cloudant access submission time values This Streams application will output the contents of the Cloudant database every two seconds. While CloudantGet is running, you can run CloudantPost at the same time and watch the database grow from CloudantGet’s console output. For more information on Cloudant’s REST API, see the Cloudant API Reference.

Using other REST APIs from SPL

Making a GET request

Using REST APIs, streams applications can integrate with nearly any web-based RESTful service.  You may use the provided samples to work with other services. To make a GET request to any RESTful service, you need the following information from the service:
  • URL to retrieve the resource
  • User name to connect to the service
  • Password to connect to the service
  • Any other service-specific information
In Streams Studio, import the CloudantGet project into your workspace. Launch the application, and specify submission time values based on the information you have collected from the service. Other services submission time values for making GET requests If the request was successful, the result will be printed to the console. If unsuccessful, an exception will display instead (see “Troubleshooting unsuccessful requests”).

Making a POST request

Similarly, to send a post request to a RESTful service, you need to provide url, username, and password of the service. In Streams Studio, import CloudantPost project into the workspace.  Build and launch the application using the appropriate values for the service. Other services submission time parameters for POST requests You may need to modify some parameters of the sample’s HTTP operators in order to integrate with different services.

Troubleshooting unsuccessful requests

  • Revisit the service’s REST API instructions. The service may expect information that wasn’t included in the request.
  • Some REST APIs require fields not supported by Streams operators. In this case, waiting for an appropriate toolkit update may be necessary

Conclusion

This article has shown how Streams applications can integrate with Cloudant and countless other services using REST APIs.

4 Comments on "Integrating with Cloudant and many other RESTful Services"

  1. Great article. There are a few gaps I am trying to fill though…

    Most of us are likely running Streams Studio QSE on a VM image. Can we launch CloudandPost from there or do we need to import it into BlueMix and run from there?

    Thanks!

  2. No matter how many times I run “spl-make-toolkit” and make clean I still get the error that the toolkit.xml is newer than sample/cloudantPost.spl.

    CDISP0127E ERROR: The following toolkit file is out of date: ./toolkit.xml. This file is newer: sample/CloudantPost.spl.
    CDISP0131E ERROR: Errors occurred while the toolkits were loading.

    Any thoughts?

  3. You should be able to launch CloudantPost from Streams Quick Start Edition.

    While investigating the “out of date” issue, I found that running the command `touch sample/CloudantPost.spl && touch toolkit.xml` from the CloudantPost directory will fix the timestamps.

    Let me know if that doesn’t resolve the issue.

Join The Discussion