The Message Hub toolkit has been developed to enable developers to quickly and easily connect Streams applications to the¬†IBM¬ģ Message Hub service.¬† IBM¬ģ¬† Message Hub for Bluemix¬ģ is a scalable, distributed, high throughput message bus based on Apache Kafka. It enables applications, either on-prem or in the cloud, to communicate with each other. The Message Hub service has proven to be a valuable tool for enabling Streams applications to communicate with each other as well as with external applications.

The toolkit handles most of the steps required to connect to the MessageHub service. This makes it easy for developers to write high-quality applications without getting bogged down by the nuances of connecting to MessageHub.  The toolkit is open source and available on GitHub.

Why It’s Needed

Until now, Streams applications have connected to the Message Hub service using the Kafka operators found in the Messaging Toolkit. While this approach works, it requires extensive configuration of the operators. This configuration typically involves having to extract portions of the Message Hub credentials JSON and create the required Kafka properties and JAAS file. This process is tedious and frustrating to deal with when something goes wrong.

The goal of the Message Hub toolkit is to abstract away as much of this configuration process as possible. By using this toolkit, developers no longer need to create a JAAS file or configure a single Kafka property. Instead, developers simply need to provide the raw MessageHub credentials JSON in either a file or an application configuration property. The toolkit will handle the process of parsing the JSON and extracting the necessary information in order to connect to the Message Hub.  All of the necessary Kafka configuration properties as well as the JAAS configuration are handled under the covers by the toolkit.


Before getting started, you will need to perform the following:

  1. Message Hub Service instance (
  2. Message Hub Credentials (found under the ‘Credentials’ section of the Message Hub service)
  3. Create a topic in the Message Hub Service. The samples use a topic called “test”, so create one in order to run the samples.
  4. Download the latest version of the toolkit from Github and unpack it.


How to Use It

The toolkit provides the MessageHubProducer and MessageHubConsumer operators to send and receive messages respectively. At a minimum, both operators require that you specify the topic and the credentials to connect to the service. There are two ways to specify your credentials Рusing an application configuration object, and using a properties file. There are two sample applications on in the toolkit that demonstrate each method.  You can look at the sample applications in the streamsx.messagehub project on GitHub.

Using an Application Configuration Object

Application Configuration objects allow you to securely store information such as credentials in your Streams instance. Applications running in the instance can then access the information stored in the object at runtime. Using an application configuration object is the recommended method to specify credentials for the operators in this toolkit.

The MessageHubAppConfigSample sample application shows how to connect to Message Hub from a Streams application using application configurations. Follow these steps to run the sample:

  1. Create the application configuration:
    Open the Streams Console and switch to the Application Dashboard
  2. On the left side of the screen, click on the Application Configuration button and then select the Application Configuration tab

  3. Click the Add Configuration button at the top
  4. Set the application configuration name to messagehub (this is important as the operators looks for this name by default)
  5. Create a new property with the following information:
    • name: messagehub.creds (again, this is important as the operators look for a property with this name)

  6. Save the application configuration.
  7. Import and run the sample:
    Download the MessageHubSamples.tar.gz archive from the github repository.
  8. Extract the archive to a directory and import the MessageHubAppConfigSample project into Streams Studio
    The SPL code for the application looks as follows:

    composite MessageHubAppConfigSample
             Message = tuple<rstring key, rstring message>
            (stream  (stream<Message> Beacon_1_out0) as Beacon_1 =  Beacon()
                    initDelay: 5f;
                    iterations : 10u;
                output Beacon_1_out0:
                    key = "key_" + (rstring)IterationCount(),
                    message = "msg_" + (rstring)IterationCount();
           //Send messages from the Beacon to MessageHub
            () as MessageHubProducer_2 = MessageHubProducer(Beacon_1_out0)
                    topic : "test";
            //Consume messages from MessageHub
             (stream<Message> MessageHubConsumer_3_out0) as
                MessageHubConsumer_3 = MessageHubConsumer()
                    topic : "test" ;
            () as Custom_4 = Custom(MessageHubConsumer_3_out0)
                    onTuple MessageHubConsumer_3_out0: 
  9. That’s it! Launch the application and it will connect to MessageHub and begin sending and receiving messages!

Note: In this example, we used the default name of “messagehub” for our application configuration object. If you specify a different name for the object when you create it, you must use the appConfigName parameter in the MessageHubConsumer or MessageHubProducer operator to specify the name of the object.

Using a properties file

If you wish to use a properties file instead of an application configuration, you will need to paste the Message Hub credentials JSON into a file names messagehub.json and store it in the applications etc/ directory. By default, the operators in will attempt to load the Message Hub credentials JSON from this file. See the MessageHubFileSample application for an example.

Default Values

The previous instructions directed the reader to add the credentials to a specific file or application configuration property. The names of the file, application configuration and property are not randomly selected. The MessageHub operators have been configured to look for the credentials in these locations by default.

The following table outlines which parameters have been configured with default values. If the user wishes to override the defaults, these are the parameters that should be set.

Parameter Name Default Value Description
appConfigName messagehub Specifies the name of the application configuration that the MessageHub operators will attempt to use to load the MessageHub credentials and the Kafka configuration properties. By default, the MessageHub operator will attempt to load properties from an application configuration named messagehub. Furthermore, the MessageHub operators have been specially configured to look for a property named messagehub.creds that contains the raw MessageHub credentials JSON.
messageHubCredentialsFile etc/messagehub.json Specifies the name of the file that contains the entire MessageHub credentials JSON. By default, this parameter will look for a file named messagehub.json in the applications etc\ directory.

Advanced Usage

The Message Hub toolkit was developed as a wrapper on top of the Kafka toolkit. This ensures that all functionality provided by the Kafka toolkit are also available in the Message Hub toolkit. To elaborate a little bit more on this, the MessageHubConsumer operator wraps the KafkaConsumer operator. All parameters provided by the KafkaConsumer operator can be used by the MessageHubConsumer operator. Likewise, the MessageHubProducer operator wraps the KafkaProducer operator, thus making all parameters available to the MessageHubProducer.

Download the Toolkit

The latest release of the MessageHub Toolkit (as well as the samples) can be downloaded from the Releases page in the streamsx.messagehub repository.

2 comments on"Introducing the Message Hub Toolkit"

  1. Hi James! Thank you for this article. I am tryin’ to follow the steps mentioned in this page to install the IBM Message Hub Toolkit. Will be great if you can answer my below question –
    1. Do we need a platform, like Eclipse to import the code mentioned in the GitHub? I am unable to proceed further as I am not getting what to do next.

    • Natasha DSilva January 29, 2018

      Hi Vinay,
      You do not need Eclipse/Streams studio to run the sample, although it makes things easier, but you can use command line. Either way, the best way to get started
      is to download the toolkit from github, and then run one of the samples. Both samples send data to Message Hub and then connect back to message hub to print the data.
      They just show different ways to specify the credentials.
      Right now the samples are included in the source release of the toolkit, so you must first
      1) Follow steps 1-3 in the “setup” section above. Copy the credentials for Message Hub from the Credentials page and save them to a file.
      2) Download and unpack the toolkit,
      3) Build and run one of the samples.

      Download the toolkit
      – Download a source release:
      Unpack it:
      tar xvf streamsx.messagehub.tgz

      Build the toolkit and the samples using gradle. First download gradle:

      Add it to your path:
      export PATH=gradle_dir/bin:$PATH
      Build the toolkit

      cd streamsx.messagehub-1.2.3/
      gradle build

      Once the toolkit build is complete, build the sample:
      From command line,
      cd toolkit/samples/MessageHubFileSample
      Paste the message hub credentials in etc/messagehub.json
      Build the sample
      gradle build

      Once the build is successful, you can run the sample in standalone mode:

      Or submit it to your instance:
      streamtool submitjob -i instance -d domain output/

      You should see output in the console or the application logs, depending on if you launched in standalone or distributed mode:


      If you have Streams 4.2 or greater, you can follow the instructions in the “Using application configuration object” section above to save your credentials to your instance and run the MessageHubAppConfigSample.

Join The Discussion