This post is an introduction to the MQ Bridge component of the IBM Message Hub service. The MQ Bridge allows you to connect IBM MQ to an instance of the Message Hub service and transfer MQ message data to a Kafka topic. If you use MQ inside your business, you can use the MQ Bridge to transport message data into the Bluemix cloud platform where it can be made available to other Bluemix services. For example: you could use the IBM Analytics for Apache Sparkâ„˘ service to run applications that analyse the MQ message data.
Since one of the best ways to learn about something is via experimentation, letâ€™s walk through using the MQ Bridge to connect to an MQ queue manager. If you want to try this, youâ€™ll need the following:
- A computer with internet access, and which has both Git and Docker installed (maybe youâ€™re using it right now to read this post)
- An IBM Bluemix account. If you donâ€™t already have one you can register to try Bluemix free for 30 days.
Hereâ€™s a diagram showing what we are going create:
Your computer will be the â€śon premiseâ€ť location where the MQ Queue Manager and Secure Gateway Clients run (both of these are Docker containers). Secure Gateway is a Bluemix service that provides a secure tunnel over the public internet. Using Secure Gateway also means that we wonâ€™t need to open up any of your computerâ€™s ports to accept inbound connections from the internet. Inside Bluemix we will use the Message Hub service to create an MQ Bridge which transfers messages from the MQ queue manager and stores the data on a Kafka topic.
Here are the steps that are needed to build this setup:
- Create a Docker container that runs an MQ queue manager, and start it on your computer.
- Create an instance of the Secure Gateway service and use it to link between the MQ queue manager and the Bluemix cloud platform. Part of this includes starting a second Docker container on your computer to run the Secure Gateway client.
- Create an instance of the Message Hub service and define an MQ Bridge that will transfer MQ message data to a Kafka topic. Link the MQ Bridge to a Secure Gateway endpoint running inside Bluemix, so it connects to the other end of the secure tunnel.
- Use sample applications to send a message to an MQ queue and verify that the data is written into a Message Hub Kafka topic.
Creating an MQ Queue Manager
MQ is already available from Docker Hub, but weâ€™re going to build our own image because we want to enable some extra options. Specifically, we want to include the sample applications inside the Docker image to simplify sending messages to an MQ queue, and also secure the queue manager in such a way that it can be accessed remotely using a particular set of credentials.
- Clone the mq-docker Github repository into a directory:
git clone https://github.com/ibm-messaging/mq-docker
The README.md for this repository contains lots of useful information. If youâ€™re interested in developing MQ applications and havenâ€™t come across the MQ Docker image before, you should definitely try it out!
- Build an MQ Docker image that includes the MQ sample applications â€“ this requires passing a few extra options on the Docker Build command line:
docker build --tag mq:mqbridge --build-arg 'MQ_PACKAGES="MQSeriesRuntime-*.rpm MQSeriesServer-*.rpm MQSeriesMsg*.rpm MQSeriesSamples*.rpm"' ./mq-docker
- Now we will create our own Docker image that extends the MQ image to enable remote access using a particular username / password pair. To do this create a file named Dockerfile with the following contents:
RUN useradd -N -G mqm -u 5001 alice && \
echo alice:passw0rd | chpasswd
COPY 20-config.mqsc /etc/mqm/
(You will probably want to change the user identifier from â€śaliceâ€ť and the password from â€śpassw0rdâ€ť)
Also note that in general, recording credentials into a Docker container is not considered a good practice, so youâ€™ll want to revisit this if you plan on using MQ inside a Docker container for a production system.
- Next create the 20-config.mqsc file which configures MQ with a channel (PASSWORD.SVRCONN) that accepts username and password credentials. Copy the following MQSC commands into this file:
DEFINE CHANNEL(PASSWORD.SVRCONN) CHLTYPE(SVRCONN) REPLACE
SET CHLAUTH(PASSWORD.SVRCONN) TYPE(BLOCKUSER) USERLIST('nobody') DESCR('Allow privileged users on this channel')
SET CHLAUTH('*') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(NOACCESS) DESCR('BackStop rule')
SET CHLAUTH(PASSWORD.SVRCONN) TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(CHANNEL) CHCKCLNT(REQUIRED)
ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) ADOPTCTX(YES)
REFRESH SECURITY TYPE(CONNAUTH)
- Now we can build our own MQ Docker image with these extra security settings, as follows:
docker build --tag mq:mqbridge-user-password.
- Finally start this image and supply some configuration values:
docker run --name="mq_mqbridge-user-password" \
-e LICENSE=accept -e MQ_QMGR_NAME=QM1 \
-p 1414:1414 -d mq:mqbridge-user-password
This will create a Docker container that runs a queue manager called QM1 that listens on port 1414. To check that the queue manager has started correctly, you can follow the log files that it creates:
docker logs --follow mq_mqbridge-user-password
Once the queue manager completes starting up, you should see the line â€śIBM MQ Queue Manager QM1 is now fully runningâ€ť at the end of the logs.
Creating an instance of the Secure Gateway service
Next up, we are going to provision an instance of the Secure Gateway service and connect it up to the MQ queue manager weâ€™ve just started. This provides secure connectivity between the Bluemix cloud platform and the system running the MQ queue manager.
- Search for the Secure Gateway service in the Bluemix Catalogue, and click on the â€śCreateâ€ť button.
- Click on the â€śAdd Gatewayâ€ť button which will launch a dialog, enter a Gateway Name of
MQand click on the â€śAdd Gatewayâ€ť button.
- Click on the â€śConnect Clientâ€ť button which will launch another dialog. Select the â€śDockerâ€ť option, which provides you with instructions on how to run the Secure Gateway client. Follow these steps â€“ and ensure that you leave the terminal window open â€“ as youâ€™ll need to come back to it to enter a command in a subsequent step.
- Find the IP address of your computerâ€™s docker0 interface:
docker network inspect bridge
This will display a JSON document â€“ you want to finds the value of the â€śGatewayâ€ť property which defaults to 172.17.0.1. Youâ€™ll need this address in the next step.
- Click on â€śDestinationsâ€ť and then on the â€śAdd Destinationâ€ť button which launches an â€śAdd Destinationâ€ť dialog. If you select the â€śAdvanced Setupâ€ť tab then you can enter all of the required information at once:
Make the following updates to the dialog:
- Ensure the â€śOn-Premise Destinationâ€ť option is selected
QM1in the â€śDestination nameâ€ť field
- Enter the IP address of the docker0 network adapter (see previous step) in the â€śResource Hostnameâ€ť field. The reason you canâ€™t use localhost here is that the Secure Gateway client runs in a Docker container that (by default) uses Dockerâ€™s bridge network mode which means that localhost from inside the container isnâ€™t the same as localhost for the system hosting the container
1414in the â€śResource Portâ€ť field
- Click on the gear icon on the destination to view information about the destination. Make a note of the value of the â€śCloud Host : Portâ€ť property. This is the endpoint to which weâ€™ll connect the MQ Bridge.
- Finally, return to the terminal where you started the Secure Gateway client and enter the following (substituting in the IP address of your docker0 adapter, as described previously):
Note that in this configuration Secure Gateway protects the data sent over the link between the Secure Gateway client and the Secure Gateway endpoint within Bluemix â€“ it doesnâ€™t protect the network links used to connect the Bridge, and the MQ queue manager to either end of the Secure Gateway tunnel. We plan on enhancing the MQ Bridge so that it is also possible to protect these links too.
Creating an instance of the Message Hub service
Now that weâ€™ve created a MQ queue manager and connected it to the Secure Gateway service, we need to connect it up with a Message Hub Kafka topic.
- Login to Bluemix, search for the Message Hub service in the Bluemix Catalogue and click on the â€śCreateâ€ť button.
- Create a Kafka topic by clicking on the â€ś+â€ť button. Enter
mqtopicas the topic name and then click on the â€śCreate topicâ€ť button. This is the Kafka topic that the MQ Bridge will send message data to.
- Create an MQ Bridge by clicking on the â€śBridgesâ€ť tab and then on the â€ś+â€ť button. Select the â€śMQ Inboundâ€ť bridge icon, then fill out the â€śCreate Bridgeâ€ť dialog with the following values, then click â€śNextâ€ť:
mqbridgeinto the Bridge Name field
mqtopicinto the Topic field
PASSWORD.SVRCONNinto the Channel field
QM1into the Queue Manager field
- Enter the â€śCloud Host : Portâ€ť value retrieved from the Secure Gateway UI (see above).
- Fill out the next page of details as follows, and click â€śCreate bridgeâ€ť:
SYSTEM.DEFAULT.LOCAL.QUEUEin the Queue Name field
- Ensure that â€śUse credentialsâ€ť is checked
- Enter the user name that you configured the MQ Docker image with in the User ID field. Earlier we used the value â€śaliceâ€ť as an example.
- Enter the password that you configured the MQ Docker image with in the Password field. Earlier we used the value â€śpassw0rdâ€ť as an example.
- Finally, youâ€™ll need to create a set of Service Credentials which you can do by clicking on the â€śService Credentialsâ€ť tab, and then on â€śView Credentialsâ€ť. This should look something like:
Make a note of the values of the â€śapi_keyâ€ť, â€śkafka_admin_urlâ€ť, and â€śkafka_broker_saslâ€ť properties. These will be used later on when we connect a sample application to Kafka to receive the data that the MQ Bridge has transferred.
Transfer Message Data using the Bridge
Right! Thatâ€™s everything setup â€“ letâ€™s use the sample applications provided by Message Hub and IBM MQ to test that we can transfer messages using the bridge. Open a terminal and do the following:
- Clone the Message Hub samples GitHub repository:
git clone https://github.com/ibm-messaging/message-hub-samples
- Create a Docker image containing the Java console sample:
docker build --tag kafka-java-console-sample message-hub-samples/kafka-java-console-sample
- Use the sample application to consume messages from the Message Hub topic that we created previously:
docker run -it kafka-java-console-sample
-consumer -topic mqtopic
Substitute in the appropriate values taken from the Message Hub credential, obtained previously. The â€śkafka_broker_saslâ€ť value should be formatted as a common separated list of host:port values surrounded by double quotes. For example:
If everything works as expected the Message Hub sample application should repeatedly print a message saying â€śNo messages consumedâ€ť. Time to send it some messages, via the MQ Bridge! To do this, weâ€™re going to use one of the MQ sample applications built in to the MQ Docker container. Open a terminal and enter the following:
- Launch a shell inside the MQ container:
docker exec -it mq_mqbridge-user-password bash -l
- Run the MQ â€śput messageâ€ť sample application and specify the name of the queue (and queue manager) that the MQ Bridge is consuming messages from:
/opt/mqm/samp/bin/amqsput SYSTEM.DEFAULT.LOCAL.QUEUE QM1
Every line of input that you type (and press enter to complete the line) into the MQ sample will be used as the payload for a new MQ message which is sent to the queue called â€śSYSTEM.DEFAULT.LOCAL.QUEUEâ€ť. Weâ€™ve configured the MQ Bridge to receive messages from this queue and send them to a Message Hub Kafka topic called â€śmqtopicâ€ť. Since weâ€™ve also configured the Message Hub samples to read from this topic and print the data from each Kafka record to the screen â€“ you should find that anything you type into the MQ sample is echoed to the screen by the Message Hub sample!
What if it doesnâ€™t work?
Hopefully you can skip over this section, but if youâ€™re having problems getting message data to flow from MQ to Message Hub Kafka, here are a few suggestions:
- It never harms to double check your work, and re-read the above sequence of steps to see if you missed something.
- The MQ Bridge outputs log information to Kibana. A link to Kibana can be found to the side of the Message Hub UI, the same UI that you used to create a Kafka topic and the MQ Bridge.
The bridge generates Kibana log records when it successfully connects to MQ. It also logs information about failed connection attempts, which often includes useful information about the reason for the failure.
- The Secure Gateway service provides troubleshooting documentation that covers resolving a number of problems that you might encounter while using the gateway.
- Look at the error logs of the MQ queue manager. These are located inside the MQ Docker container at both /var/mqm/errors and /var/mqm/qmgrs/QM1/errors
Thank you for trying the MQ Bridge. I, and the rest of the Message Hub team would love to hear about your experiences, positive or negative. You can send us feedback via the @IBMmessaging twitter account.