Introduction

This post is an introduction to the MQ Bridge component of the IBM Message Hub service. The MQ Bridge allows you to connect IBM MQ to an instance of the Message Hub service and transfer MQ message data to a Kafka topic. If you use MQ inside your business, you can use the MQ Bridge to transport message data into the Bluemix cloud platform where it can be made available to other Bluemix services. For example: you could use the IBM Analytics for Apache Spark™ service to run applications that analyse the MQ message data.

Since one of the best ways to learn about something is via experimentation, let’s walk through using the MQ Bridge to connect to an MQ queue manager. If you want to try this, you’ll need the following:

  • A computer with internet access, and which has both Git and Docker installed (maybe you’re using it right now to read this post)
  • An IBM Bluemix account. If you don’t already have one you can register to try Bluemix free for 30 days.

Here’s a diagram showing what we are going create:

Your computer will be the “on premise” location where the MQ Queue Manager and Secure Gateway Clients run (both of these are Docker containers). Secure Gateway is a Bluemix service that provides a secure tunnel over the public internet. Using Secure Gateway also means that we won’t need to open up any of your computer’s ports to accept inbound connections from the internet. Inside Bluemix we will use the Message Hub service to create an MQ Bridge which transfers messages from the MQ queue manager and stores the data on a Kafka topic.

Here are the steps that are needed to build this setup:

  1. Create a Docker container that runs an MQ queue manager, and start it on your computer.
  2. Create an instance of the Secure Gateway service and use it to link between the MQ queue manager and the Bluemix cloud platform. Part of this includes starting a second Docker container on your computer to run the Secure Gateway client.
  3. Create an instance of the Message Hub service and define an MQ Bridge that will transfer MQ message data to a Kafka topic. Link the MQ Bridge to a Secure Gateway endpoint running inside Bluemix, so it connects to the other end of the secure tunnel.
  4. Use sample applications to send a message to an MQ queue and verify that the data is written into a Message Hub Kafka topic.

Creating an MQ Queue Manager

MQ is already available from Docker Hub, but we’re going to build our own image because we want to enable some extra options. Specifically, we want to include the sample applications inside the Docker image to simplify sending messages to an MQ queue, and also secure the queue manager in such a way that it can be accessed remotely using a particular set of credentials.

  1. Clone the mq-docker Github repository into a directory:

    git clone https://github.com/ibm-messaging/mq-docker

    The README.md for this repository contains lots of useful information. If you’re interested in developing MQ applications and haven’t come across the MQ Docker image before, you should definitely try it out!

  2. Build an MQ Docker image that includes the MQ sample applications – this requires passing a few extra options on the Docker Build command line:
    docker build --tag mq:mqbridge --build-arg 'MQ_PACKAGES="MQSeriesRuntime-*.rpm MQSeriesServer-*.rpm MQSeriesMsg*.rpm MQSeriesSamples*.rpm"' ./mq-docker
  3. Now we will create our own Docker image that extends the MQ image to enable remote access using a particular username / password pair. To do this create a file named Dockerfile with the following contents:

    FROM mq:mqbridge
    RUN useradd -N -G mqm -u 5001 alice && \
    echo alice:passw0rd | chpasswd
    COPY 20-config.mqsc /etc/mqm/

    (You will probably want to change the user identifier from “alice” and the password from “passw0rd”)

    Also note that in general, recording credentials into a Docker container is not considered a good practice, so you’ll want to revisit this if you plan on using MQ inside a Docker container for a production system.

  4. Next create the 20-config.mqsc file which configures MQ with a channel (PASSWORD.SVRCONN) that accepts username and password credentials. Copy the following MQSC commands into this file:
    DEFINE CHANNEL(PASSWORD.SVRCONN) CHLTYPE(SVRCONN) REPLACE
    SET CHLAUTH(PASSWORD.SVRCONN) TYPE(BLOCKUSER) USERLIST('nobody') DESCR('Allow privileged users on this channel')
    SET CHLAUTH('*') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(NOACCESS) DESCR('BackStop rule')
    SET CHLAUTH(PASSWORD.SVRCONN) TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(CHANNEL) CHCKCLNT(REQUIRED)
    ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) ADOPTCTX(YES)
    REFRESH SECURITY TYPE(CONNAUTH)
  5. Now we can build our own MQ Docker image with these extra security settings, as follows:

    docker build --tag mq:mqbridge-user-password .

  6. Finally start this image and supply some configuration values:
    docker run --name="mq_mqbridge-user-password" \
    -e LICENSE=accept -e MQ_QMGR_NAME=QM1 \
    -p 1414:1414 -d mq:mqbridge-user-password

This will create a Docker container that runs a queue manager called QM1 that listens on port 1414. To check that the queue manager has started correctly, you can follow the log files that it creates:
docker logs --follow mq_mqbridge-user-password

Once the queue manager completes starting up, you should see the line “IBM MQ Queue Manager QM1 is now fully running” at the end of the logs.

Creating an instance of the Secure Gateway service

Next up, we are going to provision an instance of the Secure Gateway service and connect it up to the MQ queue manager we’ve just started. This provides secure connectivity between the Bluemix cloud platform and the system running the MQ queue manager.

  1. Search for the Secure Gateway service in the Bluemix Catalogue, and click on the “Create” button.
  2. Click on the “Add Gateway” button which will launch a dialog, enter a Gateway Name of MQ and click on the “Add Gateway” button.
  3. Click on the “Connect Client” button which will launch another dialog. Select the “Docker” option, which provides you with instructions on how to run the Secure Gateway client. Follow these steps – and ensure that you leave the terminal window open – as you’ll need to come back to it to enter a command in a subsequent step.
  4. Find the IP address of your computer’s docker0 interface:
    docker network inspect bridge

    This will display a JSON document – you want to finds the value of the “Gateway” property which defaults to 172.17.0.1. You’ll need this address in the next step.

  5. Click on “Destinations” and then on the “Add Destination” button which launches an “Add Destination” dialog. If you select the “Advanced Setup” tab then you can enter all of the required information at once:

    Make the following updates to the dialog:

    1. Ensure the “On-Premise Destination” option is selected
    2. Enter QM1 in the “Destination name” field
    3. Enter the IP address of the docker0 network adapter (see previous step) in the “Resource Hostname” field. The reason you can’t use localhost here is that the Secure Gateway client runs in a Docker container that (by default) uses Docker’s bridge network mode which means that localhost from inside the container isn’t the same as localhost for the system hosting the container
    4. Enter 1414 in the “Resource Port” field
  6. Click on the gear icon on the destination to view information about the destination. Make a note of the value of the “Cloud Host : Port” property. This is the endpoint to which we’ll connect the MQ Bridge.
  7. Finally, return to the terminal where you started the Secure Gateway client and enter the following (substituting in the IP address of your docker0 adapter, as described previously):

    acl allow :1414

Note that in this configuration Secure Gateway protects the data sent over the link between the Secure Gateway client and the Secure Gateway endpoint within Bluemix – it doesn’t protect the network links used to connect the Bridge, and the MQ queue manager to either end of the Secure Gateway tunnel. We plan on enhancing the MQ Bridge so that it is also possible to protect these links too.

Creating an instance of the Message Hub service

Now that we’ve created a MQ queue manager and connected it to the Secure Gateway service, we need to connect it up with a Message Hub Kafka topic.

  1. Login to Bluemix, search for the Message Hub service in the Bluemix Catalogue and click on the “Create” button.
  2. Create a Kafka topic by clicking on the “+” button. Enter mqtopic as the topic name and then click on the “Create topic” button. This is the Kafka topic that the MQ Bridge will send message data to.
  3. Create an MQ Bridge by clicking on the “Bridges” tab and then on the “+” button. Select the “MQ Inbound” bridge icon, then fill out the “Create Bridge” dialog with the following values, then click “Next”:

    1. Enter mqbridge into the Bridge Name field
    2. Enter mqtopic into the Topic field
    3. Enter PASSWORD.SVRCONN into the Channel field
    4. Enter QM1 into the Queue Manager field
    5. Enter the “Cloud Host : Port” value retrieved from the Secure Gateway UI (see above).
  4. Fill out the next page of details as follows, and click “Create bridge”:

    1. Enter SYSTEM.DEFAULT.LOCAL.QUEUE in the Queue Name field
    2. Ensure that “Use credentials” is checked
    3. Enter the user name that you configured the MQ Docker image with in the User ID field. Earlier we used the value “alice” as an example.
    4. Enter the password that you configured the MQ Docker image with in the Password field. Earlier we used the value “passw0rd” as an example.
  5. Finally, you’ll need to create a set of Service Credentials which you can do by clicking on the “Service Credentials” tab, and then on “View Credentials”. This should look something like:

    Make a note of the values of the “api_key”, “kafka_admin_url”, and “kafka_broker_sasl” properties. These will be used later on when we connect a sample application to Kafka to receive the data that the MQ Bridge has transferred.

Transfer Message Data using the Bridge

Right! That’s everything setup – let’s use the sample applications provided by Message Hub and IBM MQ to test that we can transfer messages using the bridge. Open a terminal and do the following:

  1. Clone the Message Hub samples GitHub repository:

    git clone https://github.com/ibm-messaging/message-hub-samples

  2. Create a Docker image containing the Java console sample:

    docker build --tag kafka-java-console-sample message-hub-samples/kafka-java-console-sample

  3. Use the sample application to consume messages from the Message Hub topic that we created previously:

    docker run -it kafka-java-console-sample -consumer -topic mqtopic

    Substitute in the appropriate values taken from the Message Hub credential, obtained previously. The “kafka_broker_sasl” value should be formatted as a common separated list of host:port values surrounded by double quotes. For example:

    "host1:port1,host2:port2"

If everything works as expected the Message Hub sample application should repeatedly print a message saying “No messages consumed”. Time to send it some messages, via the MQ Bridge! To do this, we’re going to use one of the MQ sample applications built in to the MQ Docker container. Open a terminal and enter the following:

  1. Launch a shell inside the MQ container:

    docker exec -it mq_mqbridge-user-password bash -l

  2. Run the MQ “put message” sample application and specify the name of the queue (and queue manager) that the MQ Bridge is consuming messages from:

    /opt/mqm/samp/bin/amqsput SYSTEM.DEFAULT.LOCAL.QUEUE QM1

Every line of input that you type (and press enter to complete the line) into the MQ sample will be used as the payload for a new MQ message which is sent to the queue called “SYSTEM.DEFAULT.LOCAL.QUEUE”. We’ve configured the MQ Bridge to receive messages from this queue and send them to a Message Hub Kafka topic called “mqtopic”. Since we’ve also configured the Message Hub samples to read from this topic and print the data from each Kafka record to the screen – you should find that anything you type into the MQ sample is echoed to the screen by the Message Hub sample!


What if it doesn’t work?

Hopefully you can skip over this section, but if you’re having problems getting message data to flow from MQ to Message Hub Kafka, here are a few suggestions:

  1. It never harms to double check your work, and re-read the above sequence of steps to see if you missed something.
  2. The MQ Bridge outputs log information to Kibana. A link to Kibana can be found to the side of the Message Hub UI, the same UI that you used to create a Kafka topic and the MQ Bridge.

    The bridge generates Kibana log records when it successfully connects to MQ. It also logs information about failed connection attempts, which often includes useful information about the reason for the failure.

  3. The Secure Gateway service provides troubleshooting documentation that covers resolving a number of problems that you might encounter while using the gateway.
  4. Look at the error logs of the MQ queue manager. These are located inside the MQ Docker container at both /var/mqm/errors and /var/mqm/qmgrs/QM1/errors

What next?

Thank you for trying the MQ Bridge. I, and the rest of the Message Hub team would love to hear about your experiences, positive or negative. You can send us feedback via the @IBMmessaging twitter account.

2 comments on"MQ Bridge in the Message Hub service"

  1. Matthew Jacobs June 27, 2017

    trying to run this step produces an error that states it can’t find the *.rpm files.

    docker build –tag mq:mqbridge –build-arg ‘MQ_PACKAGES=”MQSeriesRuntime-*.rpm MQSeriesServer-*.rpm MQSeriesMsg*.rpm MQSeriesSamples*.rpm”‘ ./mq-docker

    I looked at the location on git and I don’t see the packages listed there either.

Join The Discussion

Your email address will not be published.