Overview

Skill Level: Beginner

Beginner

Introduction Shared subscription allows one to build scalable applications which will load balance device events across multiple instances of the application by making a few changes to how the application connects to the IBM Watson IoT Platform As shown in the figure, the device events are shared between the application 1 and application 2 by […]

Ingredients

Software Requirements

  • IBM Bluemix account
  • JDK 7 or later
  • Git
  • Maven

Step-by-step

  1. Introduction

    Shared subscription allows one to build scalable applications which will load balance device events across multiple instances of the application by making a few changes to how the application connects to the IBM Watson IoT Platform As shown in the figure, the device events are shared between the application 1 and application 2 by using the shared subscription support.

    The shared subscription might be needed if one instance of back-end enterprise application can not keep up with the number of messages being published to a specific topic space, say for e.g., if many devices were publishing events that are being processed by a single application. The Watson IoT service extends the MQTT 3.1.1 specification to provide support for shared subscriptions and is limited to non-durable subscriptions only. The IoT doesn’t retain messages when the non-durable application disconnects from Watson IoT Platform.

    Refer to Scalable Applications section of the document to know more about the shared subscription support.

    This recipe demonstrates the shared subscription support by running a sample application – with and without the shared subscription support, which will help you to build scalable applications. The recipe is written for Windows platform, but it can be executed on other platforms as well.

  2. Sample Code

    Refer to this section, in case you want to know what’s happening under the hoods

    There are two samples used in this recipe to demonstrate the shared subscription support:

    • Device sample that publishes an event, every second to IBM Watson IoT Platform
    • Application sample that subscribes to all the device events in the given IBM Watson IoT Platform organization and outputs the events in real-time

    Both the samples are written using the Java Client Library for IBM Watson IoT Platform that simplifies the interactions with the IBM Watson IoT Platform.

    The Java Client Library for IBM Watson IoT Platform enables the shared subscription support based on the value set on the property “Shared-Subscription” by the user. When “Shared-Subscription” property is set to “true”, it connects the application in shared subscription mode, otherwise, connects in stand-alone mode(default behavior). In order to enable the shared subscription support or scalable application support, the library internally forms the client id as “A:org_id:app_id” and connects to IoT Platform, where

    • A indicates the application is a scalable application
    • org_id is your unique organization ID, assigned when you sign up with the service. It will be a 6 character alphanumeric string.
    • app_id is a user-defined unique string identifier for this client.

    On the other hand, application that doesn’t require the shared subscription support connects to IoT Platform with client id of the form “a:org_id:app_id”. (Note the difference in first character).

    Application(s) not making use of the Client Libraries must use the client id of the form “A:org_id:app_id” while connecting to the IBM Watson IoT Platform to enable shared subscription support. Also, multiple instances of the application(s) must connect using the same client id in order to share the load.

  3. Examples

    Example applications could be,

    • Process the device events in real-time to detect anomalies
    • Process the events in real-time to display the maximum, minimum, standard deviation and average measured value of of the datapoints (sensor readings) at any given time
    • Record the events in Cloudant NoSQL DB or any other storage in real-time – Later, this could be used for historical data analysis

    In these examples, a single instance of an application may not be able to keep the load when more and more devices start publishing the events. In which case, the shared subscription allows the application users to build and run multiple instances of the applications where the load would be shared across these applications equally.

    One could use Apache Spark and more precisely the streaming capability of Spark (Spark Streaming library) to process the events in real-time. The Apache Spark Streaming application is a best usecase for shared subscription where multiple receivers (DStreams) can be created using the same client id to share the incoming load. The application developers just need to worry about setting the right number of receivers and executors when the load increases.

  4. Register your Device in IBM Watson IoT Platform

    In order to demonstrate the shared subscription, we need to setup the IBM Watson IoT Platform organization and register devices in it. This section shows how you can setup the same.

    Carry out all the steps present in this recipe to register your device in IIBM Watson IoT Platform. When the device is registered, you will be provided with the registration details shown below. Make a note of them, we need these details to connect the device to Watson IoT Platform later.

  5. Publish Device Events

    To demonstrate the shared subscription, we need to publish the events from device(s) to IBM Watson IoT Platform registered organization. This section shows how you can publish device events

    1. Download and install Maven and Git if not installed already.
    2. Open a command prompt and clone the shared-subscription-samples project using git clone as follows,
      git clone https://github.com/ibm-messaging/iot-scalable-application-samples.git
    3. Navigate to ibmiot-shared-subscription-sample project,
      cd scalable-application-samplesjavaibmiot-shared-subscription-sample 
    4. Build the project using maven as follows,
      mvn clean package 

      (This will download the Java Client library for Watson IoT Platform, other required dependencies and starts the building process. Once built, the sample can be located in the target directory, for example, targetibmiot-shared-subscription-sample-0.0.1.jar)

    5. Modify the device.prop present under target/classes directory, file by entering the following device registration details that you noted in the previous step:
      Organization-ID = <Your Organization ID>
      Device-Type = <Your Device Type>
      Device-ID = <Your Device ID>
      Authentication-Method = token
      Authentication-Token = <Your Device Token>
    6. Run the device event publish sample by specifying the following command:
      mvn exec:java -Dexec.mainClass="com.ibm.iotf.sample.client.device.DeviceEventPublishWithCounter"

      (The sample code is present here)

    7. This code makes the device publishes events every 1 second – each event contains a counter, name, cpu and memory usage of the process as shown below:
    8. Observe that the datapoint event-count is increased for every new event that gets published.

    Let’s consume and process these events by the application(s) – with and without shared subscription feature in the following sections.

  6. Consume Events without Shared Subscription

    Consume and process the device events by a list of applications without shared subscription

    As shown in the figure below, the device events won’t be shared between the applications when they connect to IBM Watson IoT Platform without shared subscription support.

     


    Generate API Key and Token of Watson IoT Platform

    In order to connect application(s) to IBM Watson IoT Platform, you must generate the API key and token. This can be achieved by carrying out following steps present in this section – Generate API Key in Watson IoT Platform.

    Note down API Key and Authentication Token. We need these to connect to IBM Watson IoT Platform to subscribe to device events.

    Start the applications

    1. Open another Command Prompt and navigate to the ibmiot-shared-subscription-sample project, for example,
      cd scalable-application-samplesjavaibmiot-shared-subscription-sample 
    2. Modify the application.prop present under target/classes directory file by entering your Organization ID, API Key and Authentication Token of the application as follows:
      id = <Unique application id>
      Organization-ID = <Your Organization ID>
      Authentication-Method = apikey
      API-Key = <Your Organization's API Key>
      Authentication-Token = <Your Organization's Token>

      (Replace the API-Key and Token with the one that you noted earlier)

    3. Run the application sample by specifying the following command:
      mvn exec:java -Dexec.mainClass="com.ibm.iotf.sample.client.application.SharedSubscriptionSample"
    4. Observe that the application connects to IBM Watson IoT Platform with the client id of the form a:org_id:app_id (as highlighted below). The client id starting with small ‘a‘, informs the IBM Watson IoT Platform that this application is a stand-alone application and does not share the device event subscription.
    5. Also, note that the application receives the device event as and when its published by the device. i.e every 1 second and in the same order as its published by the device. (see the highlighted one in the above figure)
    6. While the first instance of the application is running, open another Command Prompt and navigate to the ibmiot-shared-subscription-sample project.
    7. Modify the application.prop file present under target/classes directory, with different application id:
      id = <Must be different from the first one>
      Organization-ID = <Your Organization ID>
      Authentication-Method = apikey
      API-Key = <Your Organization's API Key>
      Authentication-Token = <Your Organization's Token>

      (Note: If you try to connect another application with the same application id, the existing application will get disconnected from the IoT Platform. Each application must differentiate itself by providing a unique id)

    8. Run another instance of application sample by specifying the same command:
      mvn exec:java -Dexec.mainClass="com.ibm.iotf.sample.client.application.SharedSubscriptionSample"
    9. Observe that, each application receives same set of events every 1 second. As highlighted below, both the applications receive events with the same “event-count” values, 509, 510 and etc.. i.e the load is not shared between these 2 applications.

    One might need shared subscription to share the load and is demonstrated in the next section.

  7. Consume Device Events using Shared Subscription

    Consume and process the device events by a list of applications using shared subscription

    As shown in the figure below, the device events would be shared between the applications when they connect to IBM Watson IoT Platform with the shared subscription support.


    1. Open another Command Prompt and navigate to the ibmiot-shared-subscription-sample project.
      cd scalable-application-samplesjavaibmiot-shared-subscription-sample 
    2. Modify the application.prop file by entering your Organization ID, API Key and Authentication Token of the application as follows:
      id = <Unique application id>
      Organization-ID = <Your Organization ID>
      Authentication-Method = apikey
      API-Key = <Your Organization's API Key>
      Authentication-Token = <Your Organization's Token>
      Shared-Subscription = true

      (Shared-Subscription must be set to true in order to run the application in shared subscription mode)

    3. Run the shared subscription sample by specifying the following command:
      mvn exec:java -Dexec.mainClass="com.ibm.iotf.sample.client.application.SharedSubscriptionSample"

      (Note: This sample subscribes to all the device events that are registered in the given organization, so the following observations might vary if you have multiple devices registered and are sending events)

    4. Observe that the application connects to IBM Watson IoT Platform with the client id of the form A:org_id:app_id (as highlighted below). The client id starting with capital ‘A‘, informs the IBM Watson IoT Platform that this application connects in shared subscription mode and shares the load with other application(s) that connects toIBM Watson IoT Platform with same client id.
    5. Also, note that the application receives the device event as and when its published by the device. i.e every 1 second and in the same order as its published by the device (see the highlighted one in the above figure). Since there is only one instance of the application running currently, it receives all the events that are published by the device.
    6. While the first instance of the application is running, open another Command Prompt and navigate to ibmiot-shared-subscription-sample project.
    7. Run another instance of shared subscription sample by specifying the same command:
      mvn exec:java -Dexec.mainClass="com.ibm.iotf.sample.client.application.SharedSubscriptionSample"

      (Make sure the application.prop file is not modified. In order to do load balancing, same application id must be used to connect to IoT Platform )

    8. Observe that both applications share all of the events published between them, that is the load is now shared between the application instance 1 and 2. As highlighted below, the application instance 1 receives the events with “event-count” values 2119 and 2121 whereas the instance 2 receives the events with “event-count” values 2118 and 2120.
    9. Stop one instance of the application and observe that the running instance cosumes all the events published by the device as there is only one instance of the application now. Similarly if you run 3 instance of the application the load will be shared across all 3 applications.
  8. Ready to go further?

1 comment on"Building Scalable Applications in IBM Watson IoT Platform"

  1. […] tutorials on how to use the IBM Watson IoT Platform for various apps build and so many […]

Join The Discussion