In this article, you will learn how to use READ (streamsx.visualization toolkit) to rapidly create real-time visualizations from data produced by Streams applications. READ is a web application for advanced visual analytics which allows you to specify a REST endpoint as a data source and within minutes, build a real-time shareable dashboard based on that data. You can also create new types of visualizations in the READ playground. For example, here are three of the 20+ different visualizations available in READ out-of-the-box:

read_chart_horsepower_vs_weight

read_chart_sc_bars_focus

read_chart_stateful_sine

READ utilizes the Angular-nvD3 and angular-leaflet-directive libraries and supports the majority of the available charts.  This article will show you how to use READ to visualize data generated by your Streams application.  We will cover the following topics:


READ tour

Check out the 2-minute interactive tour of READ which walks you through READ’s key features. The three main parts of READ are:

  1. My Apps where data is materialized into beautiful visualizations within READ applications. A READ application is simply a collection of dashboards, datasets, and visualizations.
  2. Playground where advanced developers can experiment with various Angular-nvD3 and Angular Leaflet chart templates and define new types of visualizations.
  3. The Docs tab contains a step-by-step tutorial for learning READ.

read_tour_screenshot


Setting up the READ environment

READ is very easy to install, and you’ll be up and running in no time! The only prerequisite for READ (besides Git) is Meteor. Install Meteor by running the following Terminal command. Windows users can download the installer here.

$ curl https://install.meteor.com/ | sh

Then, download READ by cloning the GitHub repository:

$ git clone https://github.com/IBMStreams/streamsx.visualization.git

To run READ, simply execute the following commands:

$ cd streamsx.visualization/READ.develop
$ meteor npm install
$ meteor

When you see the following message, READ is ready to be accessed in your browser at http://localhost:3000/.

=> App running at: http://localhost:3000/

Various ways to connect SPL applications to READ

To create visualizations in READ, you will need to first create datasets in READ. There are three ways to turn SPL data streams into READ datasets. The following table provides a quick overview of these three methods. The time series example application in the next two sections illustrates these methods in greater detail.

Scenario What you need in SPL What you need in READ
Recommended in most situations Send the stream of data to be visualized to the HTTPTupleView operator with CORS setting enabled Create a URL dataset using a URL of the form:
http://server:port/operatorName/ports/input/portNum/tuples
For example: http://192.168.147.130:8080/currentTime/ports/input/0/tuples
You can also enable polling to periodically update the URL dataset.
Faster refresh rate for datasets and visualizations Send the stream of data to be visualized to the WebSocketSend operator Create a WebSocket dataset using a URL of the form:
ws://server:port
For example: ws://192.168.147.130:8081
A third alternative View annotation on the operator that is generating the data In order to access Streams view annotation-based data in READ, we need to set up and configure Node-RED flows (bundled as part of READ). See the appendix for detailed instructions.

Time Series Example Application: SPL Portion

We will now develop the SPL portion of the time series example application in Streams Studio. To make this tutorial more accessible, we’ll use the IBM Streams Quick Start Edition V4.2 VMware image, where Studio is installed and ready to use. First, you need version 2.8+ of the streamsx.inet toolkit. We need this toolkit in order to have access to the HTTPTupleView and WebSocketSend operators. Then, you’ll create a Streams project and an application in that project.

Follow these steps to check if you have a suitable version of the streamsx.inet toolkit, and update it if necessary:

  1. In Streams Studio, in the Streams Explorer view, expand IBM Streams Installations > IBM Streams 4.2.0.0 > Toolkit Locations > STREAMS_SPLPATH.
  2. Look for the com.ibm.streamx.inet toolkit and check the version number in [] next to it.
  3. If the version listed is older than version 2.8, update the toolkit:
    1. In Terminal, run the following commands to download the latest version of the toolkit and build it:
      $ wget https://github.com/IBMStreams/streamsx.inet/archive/master.zip
      $ unzip master.zip
      $ cd streamsx.inet-master
      $ ant
      
    2. In Streams Studio, add the toolkit location:
      • In the Streams Explorer view, right-click Toolkit Locations and click Add Toolkit Location.
      • In the Add Toolkit Location dialog, click on the Directory button to browse for the directory where the toolkit is located (i.e., the streamsx.inet-master directory from the previous step). Click OK.

To create a project and an application:

  1. Click File > New > Project. Alternatively, right-click in the Project Explorer view and select New > Project.
  2. In the New Project dialog, expand IBM Streams Studio and select SPL Application Project. Then, click Next.
  3. In the SPL Application Project dialog, enter the following information and then click Next:
    • In the Project name field, enter READ.
    • In the Namespace field, enter application.
    • In the Main composite field, enter Main.
  4. In the SPL Project Configuration dialog, ensure that the com.ibm.streamsx.inet toolkit (version 2.8+) is checked. Then, click Finish.

The code module named READ.spl opens in the graphical editor with an empty composite named Main in the canvas. Let’s switch over to the SPL Editor view by right-clicking on the canvas and selecting Open with SPL Editor. The application we’ll be using has already been written and documented in this GitHub gist. Delete the existing code template in READ.spl and copy the code in the gist into the text editor. Then, save the file.

read_studio_spl

Let’s take a look at the SPL application. It’s very simple, containing only 3 operators that provide various ways to access the data in READ:

  • A Beacon operator called time. It generates a stream of tuples containing three attributes: timestamp, sine, and cos. A view annotation has been added so that we can access the stream RESTfully through the Streams view server. We will sample 5 tuples every second, and keep the latest 60 samples within this view.
  • A HTTPTupleView operator called currentTime. It allows us to access the tuples through a simple HTTP GET API. In READ, we will poll this API at a set time interval to retrieve the latest tuple and buffer the tuples using a stateful transform dataset. The time points at which data is transferred from the SPL application to READ is under the control of READ and can be configured via the polling interval.
  • A WebSocketSend operator called latestTime. It creates an open communication channel for the real-time data transfer of tuples to READ. Unlike HTTPTupleView, the time points at which data is transferred from the SPL application to READ is under the control the SPL application.

Now, build the application to generate an application bundle and submit it to a running Streams instance. By default, any file saves will build the application automatically, so you should already see a successful completion message in the Console view. If not, we can trigger a build manually. In the Project Explorer view, select the READ project and right-click on it. From the menu, select Build Active Configurations. Ensure that the build completes successfully without error.

Then, submit the application bundle (application.Main.sab) to a Streams instance by expanding READ > application in the Project Explorer view. Right-click on the Main build configuration, and select Launch > Launch Active Build Config To Running Instance. Accept the default properties and click Launch. To check on the job’s status, switch over to the Streams Explorer view and expand Streams Instances. Then, right-click on the default:StreamsInstance@StreamsDomain item and select Show Instance Graph. You should see the three operators all colored in green, which indicates healthy status.

read_studio_instance_graph

Now that the job is running, quickly verify that the data can be accessed. Testing the REST endpoint (from the HTTPTupleView operator) is the simplest. In a browser, navigate to the following URL (replace the IP address in the URL with the IP address of the host where your Streams job is running):
http://192.168.147.130:8080/currentTime/ports/input/0/tuples
You should see a single tuple of the form:
[{"timeStamp":122557,"sine":-0.6094818845926238,"cos":-0.5138289889139901}]
When you refresh the page, a new tuple is retrieved.


Time Series Example Application: READ Portion

With the three data sources that we have defined in SPL, we have everything we need to start creating real-time visualizations in READ! The basic flow for creating content in READ is as follows:

  1. Create a READ app
  2. Create any number of dashboards in that app
  3. Create any number of datasets in a dashboard using a variety of ways, such as getting JSON data from a URL,
    getting JSON.stringify()-ed data from a WebSocket, or transforming/joining existing datasets in READ to create new ones
  4. Create any number of visualizations from a dataset to display your data. Layout these visualizations appropriately within the dashboard once you are done.

We’ll create a single app, which will contain one dashboard, six datasets (two for each source), and three visualizations (one for each source) using the following steps.

  1. Start by creating a new app. Open My Apps, and in the left sidebar, click on the read_plus_button button. In the input field, enter Time Series. In the main section, there are settings that you can configure, which will be covered in detail later in this article.
  2. Create a dashboard in your app. In the menu bar at the top, click on Dashboards next to Apps. Ensure that the correct app is selected in the left sidebar. Then, click on the read_plus_button button. In the input field, enter Sine and Cos.
  3. To create our first dataset which periodically polls and retrieves data from the SPL HTTPTupleView operator, switch to the Datasets tab. Click on the read_plus_button button and edit the dataset to match the following information:
    • Name: Time_HTTPView
    • Type: URL
    • URL: http://192.168.147.130:8080/currentTime/ports/input/0/tuples
      • Note: Update the IP address to match that of your Streams job
    • Polling: enabled
    • Polling Interval (msec): 1000
  4. Create a second dataset by transforming the first dataset, as per the following settings:
    • Name: Stateful_HTTPView
    • Type: Transform
    • Parent Dataset(s): Time_HTTPView
    • Transform Function:
      (x, s) => {                                // x is the parent dataset, and s contains the current state
          if (s.length === 30) s.splice(0, 1)    // If s already has 30 items, delete the first item
          s.push(x[0]);                          // Push the latest data point into the state array
          return s;                              // Return the state
      }
      
    • Stateful Transformation: enabled
    • Initial State: []
  5. Now that we have the data we need, we can create a visualization to easily and beautifully display this data. Navigate to Visualizations and ensure that the Stateful_HTTPView dataset is selected in the left sidebar. Click on the read_plus_button button. Fix this visualization as per the following specifications.
    • Name: HTTPTupleView
    • Chart Type: Multi
    • Basic Options:
      {
          "xKey": "timeStamp",
          "yKeys": [{
              "key": "sine",
              "type": "area",
              "yAxis": 1
          }, {
              "key": "cos",
              "type": "area",
              "yAxis": 2
          }],
      }
      
    • Advanced Options:
      {
          'chart': {
              'type': 'multiChart',
              'margin' : {
                  'top': 20,
                  'right': 40,
                  'bottom': 40,
                  'left': 55
              },
              'color': d3.scale.category10().range(),
              'useInteractiveGuideline': true,
              'duration': 500,
              'xAxis': {
                  'axisLabel': 'Time (ms)',
                  'tickFormat': function(d){
                      return d3.format(',f')(d);
                  }
              },
              'yAxis1': {
                  'axisLabel': 'Value',
                  'tickFormat': function(d){
                      return d3.format('.02f')(d);
                  },
                  'axisLabelDistance': -10
              },
              'yAxis2': {
                  'tickFormat': function(d){
                      return d3.format('.02f')(d);
                  }
              }
          }
      }
      
  6. Usage Info accompanies every visualization & dataset and provides documentation on the various aspects that we just exercised.
  7. Here comes the fun part! Let’s look at our newly created chart in the dashboard by switching to the Dashboards view. Verify that your chart looks similar to the one below. Notice that the chart has been added automatically, but it’s a bit small. Resize it by dragging the bottom-right corner to the right. Now, we can clearly see the changing data. Hovering over the chart will bring up a popup with the data values at that particular point in time.READ_HTTPTupleView
  8. Create a dataset which receives data from the SPL WebSocket operator using the following specifications:
    • Name: Time_WebSocket
    • Type: URL
    • URL: ws://192.168.147.130:8081
      • Note: Update the IP address to match that of your Streams job
  9. Create a dataset which transforms and aggregates the WebSocket data using the following specifications:
    • Name: Stateful_WebSocket
    • Type: Transform
    • Parent Dataset(s): Time_WebSocket
    • Transform Function:
      (x, s) => {
          if (s.length === 200) s.splice(0, 1)
          s.push(x.tuples[0].tuple);
          return s;
      }
      
    • Stateful Transformation: enabled
    • Initial State: []
  10. Create a line chart using Stateful_WebSocket as the dataset source:
    • Name: WebSocket
    • Chart Type: Line
    • Basic Options:
      {
          "xKey": "timeStamp",
          "yKeys": ["sine", "cos"]
      }
      
    • Advanced Options:
      {
          'chart': {
              'type': 'lineChart',
              'margin': {
                  'top': 20, 
                  'right': 40, 
                  'bottom': 40,
                  'left': 55
              },
              'x': function(d){ return d.x; },
              'y': function(d){ return d.y; },
              'useInteractiveGuideline': true,
              'xAxis': {
                  'axisLabel': 'Time (ms)',
                  'tickFormat': function(d){
                      return d3.format(',f')(d);
                  }
              },
              'yAxis': {
                  'axisLabel': 'Value',
                  'tickFormat': function(d) {
                      return d3.format('.02f')(d);
                  },
                  'axisLabelDistance': -10
              },
              'color': function (d, i) {
                  var myColors = ['#1f77b4', '#d62728'];
                  return myColors[i];
              }
          }
      }
      
  11. Verify that your chart looks similar to the one below.
    READ_WebSocket
  12. Configure Node-RED as described in this section in order to create a Node-RED URL (http://localhost:9000/streams/waves) for accessing the view annotation data.
  13. Create a dataset which receives data from Node-RED API using the following specifications:
    • Name: Time_View_Annot
    • Type: URL
    • URL: http://localhost:9000/streams/waves
      • Note: Update the IP address if necessary to match the host where you are running Node-RED
    • Polling: enabled
    • Polling Interval (msec): 1000
  14. Create a dataset which transforms and aggregates the WebSocket data using the following specifications:
    • Name: Transform_View_Annot
    • Type: Transform
    • Parent Dataset(s): Time_View_Annot
    • Transform Function:
      x => _.pluck(x.viewItems, 'data')
      

    Within the transform function above, x is the input argument and contains the data from the parent dataset. The right side of the arrow function uses the pluck function from the Underscore.js library, which allows you to easily extract the data property value from x.viewItems.

  15. Create a line chart using Transform_View_Annot as the dataset source:
    • Name: View Annotation
    • Chart Type: Line
    • Basic Options:
      {
          "xKey": "timeStamp",
          "yKeys": ["sine", "cos"]
      }
    • Advanced Options:
      {
          'chart': {
              'type': 'lineChart',
              'margin': {
                  'top': 20, 
                  'right': 40, 
                  'bottom': 40,
                  'left': 55
              },
              'x': function(d){ return d.x; },
              'y': function(d){ return d.y; },
              'useInteractiveGuideline': true,
              'xAxis': {
                  'axisLabel': 'Time (ms)',
                  'tickFormat': function(d){
                      return d3.format(',f')(d);
                  }
              },
              'yAxis': {
                  'axisLabel': 'Value',
                  'tickFormat': function(d) {
                      return d3.format('.02f')(d);
                  },
                  'axisLabelDistance': -10
              },
              'color': function (d, i) {
                  var myColors = ['red', 'green'];
                  return myColors[i];
              }
          }
      }
      
  16. Verify that your chart looks similar to the one below.
    READ_ViewAnnotation

Navigate to the Dashboards view and look at your creations! You started with a simple SPL application, retrieved the data streams, buffered the data, and transformed the data into beautiful visualizations. From here, you can pull in more complex data from a variety of sources, and build more advanced charts.

If you would like to verify your work, you can import the complete Time Series READ app into your READ instance.


Sharing your dashboards and other topics

Sharing your dashboards

You have created a number of dashboards and visualizations that are stored locally and are only accessible by you. What if you want others to gain insight from your data? There are a couple of ways to go about this:

  1. The first is make your app publicly accessible by sharing its URL. By doing so, those who access your app from the URL will only be able to view the dashboards as-is, and will not have permissions to modify the app in any way. To share your app:
    1. Navigate to the streamsx.visualization/READ.share folder in a Terminal window and run these commands:
      $ export MONGO_URL="mongodb://localhost:3001/meteor"
      $ meteor npm install
      
    2. Find the public facing IP address for your machine (for example, using the ifconfig command if you are working in a Mac or Linux environment). In my QSE environment, the IP address is 172.16.164.130. Using the IP address, run a Terminal command like the following:
      $ meteor --port 172.16.164.130:3002
      
    3. Back in READ, navigate to the Apps tab and do the following:
      • Uncheck the Private option for your application.
      • In the READ.Sharing service URL input field, enter the URL where the READ.share service is running (http://172.16.164.130:3002 in our example).
    4. Use the Public Link for App URL generated by READ to share your READ app publicly.

    When creating URL-based datasets within your READ app, ensure that you use only publicly accessible URLs (and not localhost or other local names and IP addresses).

  2. To allow others to clone your application and associated dashboards/datasets/visualizations into their own local installation of READ, export your application. Navigate to the Apps tab and click on the read_export_app button to export your app to the clipboard. Paste the JSON into a GitHub Gist, file or email and send it to the other interested parties. In their READ instance, they can click on the read_import_app button to import the app by pasting the JSON into the editor and clicking Import App. Now, they are free to enhance and/or modify the app.

Advanced visualizations

Bundled with READ is a sample application that provides a glimpse of the type of visualizations (and accompanying dashboards and datasets) that are available. The sample app illustrates how you can go about creating a variety of awesome maps, geovisualizations, sunburst charts, parallel coordinate plots, statistical plots such as box plots, and candlestick charts–all of which will be updated in real-time when their associated dataset is updated.


Contributing to READ

READ is an open source project on GitHub. We encourage you to get involved in a variety of ways.

  • Contributing code or documentation by opening pull requests
  • Submitting feature requests or bugs by logging issues
  • Sharing your READ dashboards and apps publicly which can be linked from the main READ GitHub page
  • Providing feedback

Your participation is certainly welcome! We plan to continually improve READ to make the process of creating visually appealing real-time charts as simple as possible, allowing developers to derive meaningful results from their data in a cinch.


Resources


Appendix: Setting up Node-RED to visualize Streams view annotations

In order to access Streams view annotation-based data in READ, we need to set up and configure Node-RED flows (bundled as part of READ). Follow these steps to do so:

  1. Install Node.js (you can choose the LTS version, v6.9.1 at the time of writing). To install via the yum package manager:
    $ sudo su
    $ curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
    $ yum -y install nodejs
    
  2. Start the Node-RED flows:
    $ cd streamsx.visualization/READ.develop/private/samples/nodered
    $ export NODE_TLS_REJECT_UNAUTHORIZED="0"
    $ node ../../../node_modules/node-red/red -s settings.js -u samples/
    
  3. Access the Node-RED editor in your browser at http://localhost:9000/
  4. Update the view annotation URL and credentials in Node-RED (refer to Figure 1 below; these instructions are tailored to the time series example app described in this article. If you have multiple view annotations, expose them using multiple Node-RED flows using the same method described here):
    • Switch to the tab named Streams views
    • Double-click on the node named waveview with password
    • Update the URL input field. The view annotation URL should be of the form:
      https://server:port/streams/rest/instances/instanceName/views/viewId/viewitems
      For example:
      https://192.168.147.130:8443/streams/rest/instances/StreamsInstance/views/StreamsInstance.application::Main_0.waveview/viewitems
    • Provide the correct username and password for accessing your Streams instance
    • Click the Done button
  5. Click the Deploy button
  6. The view annotation data is now available via the following Node-RED URL:
    http://localhost:9000/streams/waves
    Create a URL dataset in READ using this URL.
  7. If you are sharing your dashboards publicly, ensure that you replace localhost in the above endpoint with the appropriate publicly accessible IP address of your host

 

READ_NodeRED_Config
Figure 1: Node-RED configuration for accessing Streams view annotation data

Join The Discussion