Building an asynchronous, message-driven backbone for reactive systems

Messaging is a method of communication between software components or applications. A messaging system is a peer-to-peer facility: A component sends a message to a destination, and the recipient can receive the message from the destination.

A messaging client can send messages to, and receive messages from, any other client connected to the messaging layer, where the layer is the fabric and the engine is the instance. Each messaging client connects to a messaging engine that provides facilities for creating, sending, receiving, and reading messages.

An asynchronous message-driven backbone refers to the communication backbone of a system that uses messages as its primary style of communication.

Why is an asynchronous message-driven backbone important for reactive applications?

Messaging enables distributed communication that is loosely coupled. The sender and the receiver do not have to be available at the same time in order to communicate. The sender does not need to know anything about the receiver; the receiver does not need to know anything about the sender. Instead, the sender and the receiver need to know only which message format and which destination to use.

This asynchronous message exchange enables reactive systems to establish a shock absorber between components that ensures loose coupling, isolation, location transparency, and provides the means to delegate errors as messages. The non-blocking nature of this communication allows recipients to only consume resources while active, which ultimately leads to less system overhead.

By decoupling the components of a reactive system, we can start to implement the other reactive characteristics defined in the Reactive Manifesto: we can enable resiliency when failures occur in certain components within our application, we can enable elasticity by scaling up or down some components and not others, where necessary.

Why message-driven and not event-driven?

Interestingly, when the Reactive Manifesto was first released in 2013 (Reactive Manifesto 1.0) the four key behaviors defined for reactive systems were event-driven, scalable, resilient and responsive. In 2014, when the Reactive Manifesto 2.0 was introduced, these changed to message-driven, elastic, resilient and responsive.

This change from event-driven to message-driven was made to better reflect the core values of reactive design. Reactive systems rely on asynchronous message-passing to establish an asynchronous boundary between components. Switching from event-driven to message-driven provides clarity when considering the full set of reactive properties defined within the manifesto. The key differentiation between the two is that messages are directed, whereas events are not: a message has a clear addressable recipient while an event just occurs for others to observe.

Employing structured message exchanges enables load management, elasticity, and flow control by shaping and monitoring the message queues in the system and applying backpressure when necessary. Location transparent messaging as a means of communication makes it possible for the management of failure to work with the same constructs and semantics across a cluster or within a single host. For some applications message ordering can be important. For example, where a series of orders must be processed in the order received, or a sequence of update applied in the order received.

What tools are available to help achieve this?

There are many tools available to help you set up an asynchronous message-driven backbone for your reactive applications. The two main variations are Kafka and MQ.

Apache Kafka: A tool for event-driven and message-driven systems

Apache Kafka is an event-streaming software platform for handling realtime data feeds. It is based on a publish-subscribe messaging model and is designed to be fault-tolerant, scalable, high-throughput, and low-latency.

Although Apache Kafka can sometimes be assumed to be strictly an event-driven tool, Kafka can actually be used for both message-driven and event-driven systems. In a message-driven application that uses Kafka, messages can be sent between its components in the form of records that can be exchanged using Kafka topics. To see how you can use Kafka to set up a high performant distributed messaging infrastructure, take a look at this article: Set up a reliable, high performant distributed messaging infrastructure with Kafka.

There are many advantages to using Kafka when building a reactive system, including increased resiliency and elasticity through its features and pub/sub mechanism. Our article Where does Kafka fit in a Reactive Architecture? takes you through how Kafka can fit in reactive systems and the advantages you can gain by using Kafka.

IBM MQ: A tool for message queuing systems

MQ stands for messaging queue. It’s an abstraction layer that allows multiple processes to communicate via various models (for example, point-to-point, publish-subscribe, and so on). Message queuing systems enable applications to communicate and exchange data in a reliable and scalable way that decouples one application from another.

Message queuing systems can be either proprietary or open source, or it can be a mix of both. It might run on premise, on private servers, or on external public cloud servers. The final solution might span multiple system architectures and deployment models.

Examples of message queuing tools include:

  • IBM MQ
  • RabbitMQ
  • ActiveMQ

IBM MQ also provide many features and tools that enable it to be used successfully in reactive systems:

  • IBM MQ enables increased resiliency through its assured delivery of data (through its ability to integrate critical messages into transactional exchanges).
  • IBM MQ provides elasticity because it can horizontally scale. As the workload for a single queue manager within an MQ system increases, more queue managers can be added to share tasks and distribute the messages across them.
  • IBM MQ clusters can be used to route messages to where they’re needed intelligently.
  • IBM MQ also provides high availability of data, enabling greater responsiveness of the system, another important reactive characteristic. IBM MQ is able to do this by providing methods to replicate the data between systems and to automatically restart the queue managers that host those messages elsewhere. This, coupled with closely clustered groups of queue managers that provide the same queues and topics, enables a continuously available system.

For examples of frameworks and toolkits that use MQ systems within reactive systems, check out this Quarkus ActiveMQ connector to Reactive Messaging article or the Reactor RabitMQ documentation.

Which tool should I use?

Developers and architects often assume that these two technologies can be trivially interchanged and that they are competing technologies applicable equally to any message-driven application, this is an incorrect assumption. As is the case in many situations, once you scrape the surface and truly understand why these technologies were created, and importantly what use cases each technology is suitable for, it quickly becomes clear that enterprise messaging technologies and event streaming technologies are actually complementary technologies instead of competing technologies. Both can be used in reactive systems but careful consideration is needed in order to select the best technology for your specific use case.

To investigate further which technology (MQ or Kafka) would be the best tool to use to enable the message-driven characteristics that are required in reactive systems, check out our Why enterprise messaging and event streaming are different article.

Summary

Designing reactive systems can be challenging, but there are tools and technologies out there to help you. Both IBM MQ and Apache Kafka can be really useful technologies to use when implementing the characteristics of reactive systems to create a more asynchronous, message-driven, highly responsive application.

Next steps

To learn more about reactive systems, explore our “Introduction to reactive systems” series.

To get started using Apache Kafka, try out this tutorial: “Deploying and using a basic Kafka instance.” Then, jump into how to develop Java programs to produce and consume messages to and from Apache Kafka.

To check out IBM MQ, try out the ScratchX extension for IBM MQ. Or, to dive deeper, why not check out the IBM MQ Developer Essentials Badge learning path.