Extending the reach of streaming queues in IBM MQ

Streaming queues allow you to deliver near-identical copies of every message that comes on to a chosen queue to a second queue. You can then process these copies in any way that you'd like in order to gain insight into your system, without disrupting normal operational processing.

In the introduction to streaming queues article, streaming queues and some of their use cases were introduced. This article reviews how you can carry out some of these use cases by using the dmpmqmsg tool, which is a command line utility that allows you to copy or move the contents of a queue, or its messages, to a file or to another queue. This article also examines event streams and how streaming queues integrate with event streams.

You can use streaming queues in an analytics scenario to integrate the file containing the queue contents into a spreadsheet in order to identify trends or filter the messages by particular properties. In a test data scenario, you might use dmpmqmsg to copy the queue contents to a different queue to be worked on by another set of applications in a sandbox environment or to mutate the messages in some way to simulate errors that the applications must be able to handle and then feed them back to a queue in a test environment. For your production environment, you'll want to write your own logic, so read the command reference in the MQ Docs.

Replaying messages from a streaming queue

To replay messages from a streaming queue back to the original queue, use dmpmqmsg.

When replaying messages, you must ensure that your queue is streaming to a different queue that isn't the queue from which we will be replaying messages. For example, if you want to replay messages from DEV.QUEUE.STREAM1 to DEV.QUEUE.1, you must terminate the connection by either breaking the connection between these queues temporarily, or connecting DEV.QUEUE.1 to another streaming queue such as DEV.QUEUE.STREAM2.

You can replay messages in several ways.

Example 1: Replaying all messages from the streaming queue only once

To put all messages from one queue DEV.QUEUE.STREAM in to a second queue DEV.QUEUE.1, use this command

dmpmqmsg -I DEV.QUEUE.STREAM -m QM1 -o DEV.QUEUE.1

The messages will be destructively consumed from the streaming queue and put onto the original queue.

If you want to achieve this outcome without destructively consuming from the streaming queue, you can browse instead:

dmpmqmsg -i DEV.QUEUE.STREAM -m QM1 -o DEV.QUEUE.1

All messages from DEV.QUEUE.STREAM will be browsed instead of destructively consumed, and they will still be copied to DEV.QUEUE.1.

Example 2: Replaying specific messages from the streaming queue based on messageID

To get a specific message from the queue based on a MessageID (m), run the following command. Since the message ID is in hex, we add the x.

dmpmqmsg -I DEV.QUEUE.STREAM -m QM1 -o DEV.QUEUE.1 -g xm<messageID>

You might also want to get specific messages based on other IDs:

• CorrelationID (c)
• GroupID (g)

In this example, an application might consume a message from DEV.QUEUE.STREAM but hit an error while processing it. Once the application has fixed the issue, you can replay that specific message and resume normal processing.

To avoid this scenario, it's good practice to use local transactions when exchanging messages so that, in the event of an error in message consumption, the message would simply be rolled back to the original queue, ready for re-consumption later on. However, if the message cannot be processed for a recurring reason, it will be put on to a backout queue.

Streaming queues and event streams

Streaming queues provide a useful way to integrate with IBM Event Streams, which will allow you to analyze real-time data from applications. This Event Streams integration tutorial explains how MQ messages can be surfaced on Kafka topics, where they can be used as a source of events for event processing.

In this example, the queue engaging in messaging is the COMMANDS queue, and messages from this are streamed to the streaming queue, COMMANDS.COPY. This streaming queue acts as the source for the Kafka connector, and the messages are fed to an events dashboard for processing.

Summary and next steps

In this article, we explored how we can use the dmpmqmsg tool as a way of taking messages from streaming queues for further analysis, reviewing a more specialized streaming queues use case - message replay.

By streaming your messages to a topic, instead of a queue, you can fan out the streamed messages to multiple subscribers – letting you do everything we've discussed here, and more, at once! See the streaming messages to topics tutorial to learn exactly how to do it!