IBM has recently announced IBM Event Streams for on–premise deployments, building on the experience of the IBM–managed service provided in the IBM Cloud. As clients explore the capabilities the technical and enterprise architecture teams are interested in understanding when it is suitable to use IBM MQ‘s publish/subscribe capabilities, and when IBM Event Streams is required.¬†
As discussed in a previous blog, IBM MQ and Event Streaming technology have several differentiating capabilities, however the publish/subscribe capabilities do on initial review appear to blur the line.¬† In reality as with many technical decisions, if you try hard enough you can use a lot of different products to solve a problem, but it doesn’t mean it is a good idea. To try and assist clients with understanding the technical considerations, the following four characteristics should be considered:
- Event History: does the solution need to be able to retrieve historical events either during normal and/or failure situations. Within a traditional publish/subscribe model, the event is published to a subscriber, once it is received it is the subscriber‘s responsibility to store this information for the future if needed. There are certain situations where the publish/subscribe model can retain the last publication, but it is certainly unusual to get the messaging solution to store historical events. For IBM Event Streams, storing event history is fundamental to the architecture, the only questions are how much and for how long. In many use cases it is critical to store this history, while in others it may be undesirable from a system resources and security point of view.¬†
- Fined Grained Subscriptions: when a topic is created in IBM Event Streams this creates one or more partitions within the solution. This is a fundamental architectural concept within the underlying Apache Kafka technology, and provides the capability to scale the solution to handle a massive amount of events. Each partition uses up resources and it is normally advisable to limit the number of topics to hundreds or maybe thousands within a single cluster. MQ publish/subscribe has a more flexible mechanism, where the topic can be a hierarchical structure such as /travel/flights/airline/flightNumber/seat, allowing more subscription points. This allows subscribing applications to select the events at a finer granularity. In addition MQ publish/subscribe selectors can be used to further refine the events of interest. For applications subscribing, it means that they are far less likely to receive events that are irrelevant to them in the case of MQ publish/subscribe, while an Event Streaming application that wants only a small proportion of the events will likely need a discarding filter to be applied early in the processing.
- Scalable Consumption: if 100 consumers subscribe to all events on a topic, a technology such as IBM MQ will create 100 messages for each published event (with the exception of multicast Pub/Sub). Each of these will be stored and, if required, persisted to disk using system resources . In the case of IBM Event Streams, the event is written once and each consumer has an index corresponding to where they are in the event history. IBM MQ is a highly scalable solution, so depending on the number of events emitted by the publisher, and the number of subscribers, this may or may not be a factor in deciding the most appropriate technology.
- Transactional Behavior: both IBM MQ and IBM Event Streams provide transactional APIs to process events, however the flexibility and capabilities provided by IBM MQ is wider and more mature than that provided by IBM Event Streams. Often in a publish/subscribe solution, the hardened transactional behavior of IBM MQ is not as critical as in a messaging queuing scenario. For further information on the transactional considerations please consult:
Using the above four characteristics should allow you to determine if a messaging Publish/Subscriber or Event Streams solution is most appropriate.¬†