Unlocking the potential of message queues

Message queues are a fundamental building block of messaging, and they are essential to many enterprise messaging solutions. They are everywhere, ensuring information exchange happens seamlessly, but it’s easy for developers to take for granted just how flexible they are!

In this article, we’ll show the versatility of message queues by exploring a range of message exchange patterns that you can achieve using a single queue, and we’ll explain when developers might want to use each exchange pattern.

Point-to-point messaging pattern

First, let’s look at the simplest messaging pattern, point-to-point messaging. Point-to-point messaging involves a single producing application putting a message on a queue for a single consuming application to get, as illustrated below.

Critically, the producer and consumer don’t need to know anything about each other; they just need to know the name of the queue that they’ll use to exchange information. Using queues to wire together single application instances might be useful to model how a set of services are joined together in a workflow, but how does this pattern scale to serve multiple application instances?

Filtering techniques for single queue exchanges

There are many ways that we can exchange messages between multiple producers and consumers using a single queue, but before we dive into those, let’s take a look at some filtering techniques that we can employ to ensure these exchanges happen efficiently.

Message selectors

Using application-defined property values that are contained in the header of a message, an application can use selectors to filter messages to get those fulfilling a specific criterion.

An example message selector would look like this:

"JMSType = 'car' AND colour = 'blue' AND weight > 2500"

This message selector will select all messages with a message type of car, the color of blue, and a weight greater than 2500 kg. In this case, JMSType is being used to define the type car, and you can find out more about this and other property names in the MQ docs. This subset of messages is then made available for the application to consume FIFO.

To get a single specific message, maybe a reply to a message sent previously, you can also select by correlationID or messageID. When you use message selectors in JMS, there are some restrictions to be aware of, which you can read about in the MQ docs.

Message priority

An application can assign priority to a message when it is put on to a queue using the Priority field in the message descriptor. Message queues will be maintained in either strict FIFO or FIFO within priority sequence, depending on the queue settings.

When a message arrives on a queue, it is inserted immediately after the last message that has the same priority. Applications can either get the next highest priority message or get a particular message based on selection criteria.

When it comes to transactions, a whole unit of work could have the same priority, meaning each message would be consumed in the order it is created. If messages have different priorities, each message would be consumed in priority order within the unit of work.

Message priority values can be between 0 and 9, with 9 being highest priority. You can read more about message priorities in the MQ docs here.

Exchange patterns

We will review these message exchange patterns:

• Many-to-many
• One-to-many (collaborative consumption)
• Many-to-one

Many-to-many

It’s a common misconception that a queue can only be used by a single producing and consuming pair, when in fact, you can have several producers and consumers all connected to the same message queue. The filtering techniques we’ve just looked at make it easy to do this!

As before, producers put their messages to this single queue to be consumed by a corresponding consumer. Consuming applications can then choose to get a particular message based on a property. For example, one consumer might get messages that have currency equal to "Euros," while another might get messages where currency is "Pounds." Consumers might also choose messages based on priority, where one consumer might only get messages with priority 9, while another processes all other messages as they arrive.

One-to-many (collaborative consumption)

Imagine you have an application producing a stream of high-value messages that need to get processed. The ability to connect multiple consumers to the queue allows the end-to-end solution to scale and deal with spikes in activity from the producer. While one application instance is busy processing a message, other consumers connected to the queue continue to respond to messages as they arrive.

To build a resilient solution, you will want to have multiple consumers available to get messages, so that even if one goes down, there are still others available. This one-to-many or collaborative consumption pattern involves several consumers to balance workload from producers connected to a queue. The producing application puts a message to the queue, to be received by only one of the consuming applications, since messages are consumed destructively from the message queues.

The message will be received by the consumer that connected to the message queue most recently for efficiency. You can read more about this pattern’s behavior in this IBM Support document.

This configuration works well when all the consumers are equally suitable for receiving and processing the messages from the producer. If, however, you want certain consumers assigned to getting certain types of messages, you can implement one of the filtering methods described earlier in this article. You can even build on this pattern further by having multiple consumers assigned to the same selection criteria, which will result in sub-groups of collaborative consumers.

Many-to-one

We can also have many producers that put work on to a single messaging queue to be processed by a single consumer. As before, this consumer might then be able to process all the load from the producer or chose to get specific messages based on properties or priority.

The applications of this exchange pattern are more niche, but one possible use case would be for a batch processing task such as taking stock of produce sold in a shop to update a database.

Again, when combined with selection techniques, you can have consumers specialized to process different types of messages.

Message Browsing

So far, we’ve looked at some of the ways we can put and get messages to and from a single queue – but there is more to queues than this.

Message browsing allows you to preview the contents of messages on a queue without destructively consuming them, so you can make decisions on how to process messages based on the payload. Maybe you want to filter a stream of messages coming in from multiple producers or want to verify the contents of particular messages in an audit. Developers use message browsing for more specialized message retrieval.

Simple browsing

The process works in a similar way to the typical "getting" of messages, in that the application must open a queue, only this time for browsing instead of getting. This will place a logical "browse cursor" before the first message in the queue. The browse cursor is a pointer that tracks the next message to browse. The application can choose to preview the first message in the queue, or the first message which satisfies a particular selection criterion. The application can then step through each message sequentially.

If a message is part of an uncommitted unit of work, it won’t be visible to the browse cursor until the whole unit of work has been committed. If you browse a message and decide you want to retrieve it, you can do so as described in the MQ docs.

Browsing and collaborative consume

Browsing can be used to enhance the "collaborative consume" exchange pattern discussed earlier in this article.

Multiple consumers can browse messages on the same queue. They will have their own browse cursor and will, by default, be able to work through the messages independently. The consumers can also mark messages that they have browsed and inform the other consumers that they have been previewed, preventing the repeated browsing and attempted retrieval of messages. So that the application is aware of the messages marked by the other consumers it’s collaborating with, it needs to open the queue using MQOO_CO_OP, which you can read more about here.

This functionality provides another way to assign messages on a queue to a particular consumer, but this time it is based on the contents of the message instead of message properties. Messages that have been marked for some time can be automatically unmarked by the queue manager so that they are once again made available to be browsed or consumed.

So far for message browsing, we’ve looked at MQI, but what about JMS? JMS applications can also browse messages in a queue and can create a separate session object to browse and then to get messages. When it comes to collaborative browsing, however, JMS consumers connected to the same queue won’t be aware of the messages browsed by each other. This means that the application designer needs to add logic to ensure that when the application attempts to browse or get a message, the message is actually available. Cases when the message may not be available include: being browsed by another consumer, being consumed by another consumer, or the expiry time elapsed.

Summary and next steps

In this article, we looked at all the exchange patterns we can achieve by using just one message queue and when they might be useful. These exchange patterns can be extended further by using message selectors. We also looked at the message browsing functionality and how it can be used to enhance the collaborative consumption exchange pattern.

It’s easy to take queue fundamentals for granted, but hopefully you now appreciate the wide variety of exchanges that they facilitate. To learn more about message queues and IBM MQ, and even earn a digital credential, head to this MQ Developer Essentials learning path!