Introduction to streaming queues

You already use message queues in IBM MQ to exchange messages between connected applications. Once a message is processed successfully, it is removed from the queue by the consuming application.

This messaging pattern is great for ensuring a consistent processing state between applications, but what if we want to gain insight into the work being carried out over the queues in our system? Streaming queues allow you to do even more with the messages flowing through your IBM MQ solution.

In this article, we'll look at what streaming queues are and how they allow you to build more sophisticated messaging solutions.

What is a streaming queue?

Streaming queues allow you to deliver near-identical copies of every message that comes on to a chosen queue to a second queue. You can then process these copies in any way that you'd like to gain insight into your system, without disrupting normal operational processing. Crucially, you can take advantage of this feature without making changes to your application code, because streaming queues can be configured with simple MQSC or from within the IBM MQ Console.

Streaming queue configurations

Streaming queues can operate in one of two modes:

• Best Effort. In this mode, the delivery of the original message will occur even if the streamed message cannot be delivered to the streaming queue (for example, if it was full at the time)
• Must duplicate. This mode ensures that either the original and the streamed message are both delivered or not at all. If the streamed message can't be delivered, then the original won't be delivered either.

As mentioned earlier, streamed messages will be a near-identical copy of the message. This is because all attributes are copied across (including descriptor fields, message ID, and correlation ID) except the message expiry attribute. This attribute is removed and either set to MQEI_UNLIMITED or to the value of CAPEXPRY set on the streaming queue. It is recommended to have some value set for CAPEXPRY, since the streaming queue could become full if there isn't an application consuming the streamed messages regularly.

Alternative destinations for streaming queues

Streaming queues don't have to be local queues. It is also possible to stream messages to remote queues and alias queues. Streaming to a remote queue allows the message to be sent to a queue hosted on another queue manager.

By streaming messages to an alias queue the copy of the message can also be sent to a topic. This means that a copy of the duplicated message will be delivered to each subscriber to the topic.

Messages sent to a topic will adhere to publish/subscribe rules though, meaning they won't be identical and will differ in the following ways:

• Having a new message ID.
• Having a generated correlation ID, depending on the configuration of the subscription.
• The UserIdentifier field will be set to the user the queue manager is running as, not the user who put the message.
• The PutApplName will show the name of the queue manager, not the name of the putting application.

Typical use cases for streaming queues

So now that we know what streaming queues are, let's look at some ways you might use them.

Analysis on data going through a system

Streaming queues serve as a record of messages that have been put onto a queue, and therefore the data that has been processed over that queue. You might choose to have an application consuming the messages off of a streaming queue to perform analysis. This could include identifying trends and anomalies or filtering out messages that you aren't concerned with right now.

An example of this would be in a supermarket. You might analyze the frequency of purchasing certain items to know when to increase the price or when to issue coupons for these items to specific customers. Another example would be a fraud detection scenario. If suspicious activity is detected, you can analyse the whole interaction to investigate.

Message replay

Streaming queues facilitate message replay, which allows you to replay messages back into the original queue to be reprocessed. This might be useful if the consuming application incorrectly consumed messages and needs to reattempt consumption.

For example, an application might remove the messages from the queue before it has completed its processing and then crash. When the application restarts, the messages are not there to be reprocessed, so you will need to replay them. In this scenario, you’ll rarely need to consume all of the copied messages, so it’s critical that you put an expiry on the copy to prevent the streaming queue filling.

Test data set

Since messages streamed to a streaming queue are from real world exchanges, the messages could be a useful source on which to base test data sets. You can collect the data from real world interactions, such as a stream of banking transactions, and then feed these into a test system as sample data.

Sending copies to multiple systems using a topic

By streaming your messages to a topic via an alias queue, you can distribute the interactions carried out over a particular queue to many different systems. This means you can have multiple copies of the streamed messages going to any combination of the systems we discussed here.

Summary and next steps

In this article, we looked at what streaming queues are and how they enable you to gain more insights into the work being carried out over your queues, as well as some of the ways you might use this information. Now head over to this tutorial to learn how you can set up a streaming queue yourself!