Introduction

The Message Hub toolkit has been developed to enable developers to quickly and easily connect Streams applications to the IBM® Message Hub service.  IBM®  Message Hub for Bluemix® is a scalable, distributed, high throughput message bus based on Apache Kafka. It enables applications, either on-prem or in the cloud, to communicate with each other. The Message Hub service has proven to be a valuable tool for enabling Streams applications to communicate with each other as well as with external applications.

The toolkit handles most of the steps required to connect to the MessageHub service. This makes it easy for developers to write high-quality applications without getting bogged down by the nuances of connecting to MessageHub.  The toolkit is open source and available on GitHub.

Why It’s Needed

Until now, Streams applications have connected to the Message Hub service using the Kafka operators found in the Messaging Toolkit. While this approach works, it requires extensive configuration of the operators. This configuration typically involves having to extract portions of the Message Hub credentials JSON and create the required Kafka properties and JAAS file. This process is tedious and frustrating to deal with when something goes wrong.

The goal of the Message Hub toolkit is to abstract away as much of this configuration process as possible. By using this toolkit, developers no longer need to create a JAAS file or configure a single Kafka property. Instead, developers simply need to provide the raw MessageHub credentials JSON in either a file or an application configuration property. The toolkit will handle the process of parsing the JSON and extracting the necessary information in order to connect to the Message Hub.  All of the necessary Kafka configuration properties as well as the JAAS configuration are handled under the covers by the toolkit.

Setup

Before getting started, you will need to perform the following:

  1. Message Hub Service instance (https://ibm.biz/BdjB7r)
  2. Message Hub Credentials (found under the ‘Credentials’ section of the Message Hub service)
  3. Create a topic in the Message Hub Service

How to Use It

The toolkit provides the MessageHubProducer and MessageHubConsumer operators to send and receive messages respectively. At a minimum, both operators require that you specify the topic and the credentials to connect to the service. There are two ways to specify your credentials – using an application configuration object, and using a properties file. There are two sample applications on in the toolkit that demonstrate each method.  You can look at the sample applications in the streamsx.messagehub project on GitHub.

Using an Application Configuration Object

Application Configuration objects allow you to securely store information such as credentials in your Streams instance. Applications running in the instance can then access the information stored in the object at runtime. Using an application configuration object is the recommended method to specify credentials for the operators in this toolkit.

The MessageHubAppConfigSample sample application shows how to connect to Message Hub from a Streams application using application configurations. Follow these steps to run the sample:

  1. Create the application configuration:
    Open the Streams Console and switch to the Application Dashboard
  2. On the left side of the screen, click on the Application Configuration button and then select the Application Configuration tab

  3. Click the Add Configuration button at the top
  4. Set the application configuration name to messagehub (this is important as the operators looks for this name by default)
  5. Create a new property with the following information:
    • name: messagehub.creds (again, this is important as the operators look for a property with this name)
    • value: <YOUR_MESSAGEHUB_CREDS_JSON>

  6. Save the application configuration.
  7. Import and run the sample:
    Download the MessageHubSamples.tar.gz archive from the github repository.
  8. Extract the archive to a directory and import the MessageHubAppConfigSample project into Streams Studio
    The SPL code for the application looks as follows:

    composite MessageHubAppConfigSample
    {
        type
             Message = tuple<rstring key, rstring message>
        graph
            (stream  (stream<Message> Beacon_1_out0) as Beacon_1 =  Beacon()
            {
                param
                    initDelay: 5f;
                    iterations : 10u;
                output Beacon_1_out0:
                    key = "key_" + (rstring)IterationCount(),
                    message = "msg_" + (rstring)IterationCount();
                    
            }
    
           //Send messages from the Beacon to MessageHub
            () as MessageHubProducer_2 = MessageHubProducer(Beacon_1_out0)
            {
                param
                    topic : "test";
     
            }
     
            //Consume messages from MessageHub
             (stream<Message> MessageHubConsumer_3_out0) as
                MessageHubConsumer_3 = MessageHubConsumer()
            {
                param
                    topic : "test" ;
            }
    
            () as Custom_4 = Custom(MessageHubConsumer_3_out0)
            {
                logic 
                    onTuple MessageHubConsumer_3_out0: 
                    {
                        println(MessageHubConsumer_3_out0);
                    }
           }
     }
  9. That’s it! Launch the application and it will connect to MessageHub and begin sending and receiving messages!

Note: In this example, we used the default name of “messagehub” for our application configuration object. If you specify a different name for the object when you create it, you must use the appConfigName parameter in the MessageHubConsumer or MessageHubProducer operator to specify the name of the object.

Using a properties file

If you wish to use a properties file instead of an application configuration, you will need to paste the Message Hub credentials JSON into a file names messagehub.json and store it in the applications etc/ directory. By default, the operators in will attempt to load the Message Hub credentials JSON from this file. See the MessageHubFileSample application for an example.

Default Values

The previous instructions directed the reader to add the credentials to a specific file or application configuration property. The names of the file, application configuration and property are not randomly selected. The MessageHub operators have been configured to look for the credentials in these locations by default.

The following table outlines which parameters have been configured with default values. If the user wishes to override the defaults, these are the parameters that should be set.

Parameter Name Default Value Description
appConfigName messagehub Specifies the name of the application configuration that the MessageHub operators will attempt to use to load the MessageHub credentials and the Kafka configuration properties. By default, the MessageHub operator will attempt to load properties from an application configuration named messagehub. Furthermore, the MessageHub operators have been specially configured to look for a property named messagehub.creds that contains the raw MessageHub credentials JSON.
messageHubCredentialsFile etc/messagehub.json Specifies the name of the file that contains the entire MessageHub credentials JSON. By default, this parameter will look for a file named messagehub.json in the applications etc\ directory.

Advanced Usage

The Message Hub toolkit was developed as a wrapper on top of the Kafka toolkit. This ensures that all functionality provided by the Kafka toolkit are also available in the Message Hub toolkit. To elaborate a little bit more on this, the MessageHubConsumer operator wraps the KafkaConsumer operator. All parameters provided by the KafkaConsumer operator can be used by the MessageHubConsumer operator. Likewise, the MessageHubProducer operator wraps the KafkaProducer operator, thus making all parameters available to the MessageHubProducer.

Download the Toolkit

The latest release of the MessageHub Toolkit (as well as the samples) can be downloaded from the Releases page in the streamsx.messagehub repository.

Join The Discussion