In this article we’ll explore the world of messaging APIs and protocols. We’ll identify why developers depend on them and focus on the capabilities of some of the most commonly used messaging APIs and protocols.

What are protocols?

The term protocol is often used in news reporting to formalize interactions between officials and heads of state. It defines a set of rules and conventions that structure human interactions during talks and negotiations. Successful outcomes depend on all parties adhering to the correct protocol(s). Get something wrong and talks and can break down, with focus shifting quickly away from a shared goal and onto the frantic efforts needed to recover the situation.

In computing, we use the term protocol in much the same way. A protocol is just a set of rules that computers follow, allowing them to interact with each other, and with the outside world, in some predictable, deterministic way.

At the lowest level, multiple protocols are being employed right now to transmit this text to your browser. Most developers never need to worry about this low-level stuff, and most of the time it is completely transparent to the end user. This is only because both ends have agreed – or maybe assumed – that a particular convention is being followed to share data. As in this case, your browser was expecting HTTP or HTTPS protocol over a TCP/IP socket to deliver some HTML-encoded data that creates some human-readable web content.

This baseline of dependable data exchange is essential to developers. We want to focus on adding value to our users and getting that value deployed quickly. Protocols like TCP/IP and TLS are ubiquitous, so unless developers have very specific requirements, it’s unlikely that any developer would want to start coding from the ground up.

Moreover, developers don’t like to keep solving the same problem again and again. It’s an anti-pattern to good application development as it distracts from adding business value and leads to increased risk of defect escape. Movements such as the Reactive manifesto help developers deliver robust, responsive solutions by promoting good application design and loosely coupled asynchronous integration.

In simple terms, messaging protocols define a structured way for applications to exchange payloads of data between applications. They also describe how an implementation should process the messages, prioritize the messages, and route data between producers and consumers. Many messaging protocols go much further in defining security concepts and helping implementers achieve availability and scale of the messaging layer in standard and consistent way. As an example, the MQTT protocol is a messaging protocol, developed as an open OASIS standard and recognized as an international standard ISO/IEC 20922:2016.

Enterprise messaging

Enterprise messaging is simple. It’s just the wiring between the different parts of a business (manufacturing, ordering, invoicing, and so on), moving data and files securely to where they need to be. Enterprise messaging is the common denominator in integrating applications seamlessly, regardless of systems or architecture.

Graphic showing how messaging is the central spoke of enterprises

Developers are measured on adding business value to the organizations they work for. For example, an airline will want to prioritize delivering delightful customer experience to set themselves apart from the competition. While smart data processing is essential to that goal, the low-level mechanics of how the data is transported is just what makes it possible. To avoid all the overhead of developing a bespoke solution, a solution architect will select a pre-existing, proven messaging technology based on a set of requirements that meet the needs of the business. This messaging technology promotes greater agility and enables the airline to align their developer resource clearly on the business outcomes required.

Enterprise messaging technologies present developers with an intuitive API and a defined protocol that they can just pick up and incorporate into their solution. It’s essentially a contract between developer code and the chosen messaging technology. Rather than worrying how transactions, message recovery, or persistence is implemented, the developer just uses the correct API and structures their code around a messaging technology that provides the quality of service (QoS) that they need.

graphic showing the characteristics of messaging: comms, security, trasnactions, persistence, scale, and recovery

Enterprise messaging protocols

One drawback with abstraction is that the world of enterprise messaging can seem daunting to new users, especially given the vast number of standards, APIs, and protocols, that are available: AMQP, JMS, MQTT, QPid, AsyncAPI, and STOMP, just to name a few. It appears to be a sea of acronyms, and we haven’t even discussed the proprietary messaging systems specifics.

Prior to explaining each of these acronyms, we need to define a clear definition for an API and protocol:

An API is the interface that an application developer codes against to interact with the messaging system and its components, for example a queue, topic, or subscription.
A protocol is sometimes referred to as the wire protocol, and it describes the format of the bytes that are transferred over the wire, and the combination of messages, or packets, that are exchanged for a particular operation.

There is a clear delineation of responsibilities between an API specification and a protocol specification. The API specification does not define the format or messages to be exchanged over the wire, whereas the messaging protocol has no control or influence over the API that is used by the application developer.

The glue between the API and the protocol is a messaging client, which will implement a given API using a particular protocol.

Let’s circle back to that list of acronyms mentioned earlier and associate these with the three categories:

APIs: JMS and proprietary APIs are provided by nearly all enterprise messaging providers, such as IBM MQ, ActiveMQ, or RabbitMQ.
Protocols: AMQP, MQTT, STOMP and the native proprietary protocols.
Clients: QPid and other proprietary-based clients.

Some of these acronyms are also overloaded, which makes the situation even more confusing. For example, QPid is an open source Apache project for a collection of capabilities for AMQP. It includes clients, server components, and tooling. One last acronym not covered is AsyncAPI, which the name makes one think that this is a messaging API, but it does not fit into the definition above, as AsyncAPI is a description of a service or API that is exposed using an asynchronous protocol, and that is deliberately agnostic of the messaging API, protocol, and client.

Choosing your enterprise messaging APIs and protocols

We are now ready to discuss each of these enterprise messaging APIs and protocols, and why they were created:

APIs
- JMS (Java Message Service): This API is the oldest open standard within enterprise messaging. It was created in the late 1990’s to provide a common Java interface across multiple messaging providers. The goal was twofold:
  - Common Skills: An application developer can learn the JMS API and interact with any messaging system that provides support for it, and any nuances are abstracted away.
  - Portability: If a company decided to move from one messaging system to another, there would be minimal application changes required. Like all standards and approaches that promote portability, it certainly lowers the effort to move, but nuances will always find their way into the application code.
- Proprietary APIs: Most messaging providers also provide their own APIs for access, especially for non-Java languages where JMS is not applicable. Interestingly, JMS has been so successful that we have seen JMS-like APIs popping up in non-Java clients, but these have often failed to gain industry-wide traction. For example, XMS exists for .NET applications in IBM MQ, using a similar interface to JMS.
Protocols: Because of the vast number of protocols, and to avoid any implied order, we’re going to discuss them based on the date they were created.
- Proprietary Protocols: It should come as no surprise that proprietary protocols existed prior to any standards. Products such as IBM MQ existed for many years prior to protocol standardization efforts. These protocols are inherently optimized for the messaging system, and normally represent the most performant approach.
- MQTT (1999): IBM created this protocol and designed it for environments with restricted bandwidth. The messaging protocol itself only requires a small footprint. MQTT became increasingly popular due to the explosion of the Internet of Things (IoT). To minimize resources, MQTT is an efficient wire protocol. There are several releases of MQTT, with the latest (V5) released in 2019. MQTT is now an open OASIS standard and recognized as an international standard ISO/IEC 20922:2016.
- AMQP (2003): Created by JPMorgan Chase, this protocol provides improved interoperability between messaging systems. Like MQTT, the protocol is a structured wire protocol, but it is a more general purpose protocol, providing features that are not available on MQTT, such as improved transactional and security support. There have been several releases with the most popular being AMQP 0-9-1 (2008) and 1.0 (2012). Although these two versions share the same protocol name, they are substantially different, which has caused frustration and confusion as messaging systems support different versions.
- STOMP (2005): This messaging protocol was inspired by HTTP and provides a human-readable messaging protocol. The human-readable decision has been a factor in its lack of adoption, due to messaging systems commonly transforming from the STOMP protocol to an internal representation. This has commonly caused non-desirable performance characteristics and therefore it has not gained substantial traction in the market.

Let’s summarize when to use which, by considering different factors:

Performance: Every messaging system will have a “native protocol”, this is commonly the first protocol that was supported. If you desire the “best” performance these two are normally the same. Therefore, in IBM MQ this would match the MQ proprietary protocol, or in the case of RabbitMQ an old version of AMQP 0-9-1.
Lightweight: MQTT is specially designed to limit the bandwidth and footprint requirements. If this is the overriding factor, then normally MQTT is the preferred approach if it provides all the capabilities required.
Feature Rich: The messaging systems “native protocols” are commonly the first to expose the latest features and capabilities, and therefore have the greatest range.
Messaging systems interoperability: To avoid vendor lock in, often AMQP is chosen due to its feature rich set of capabilities, and wide adoption by messaging systems. As mentioned previously, although it lowers the barrier, it does not remove all barriers as vendor specifics bleed through the wire protocol as characteristics of the implementation.

It is worth highlighting that the above list of factors represents the common options, but they really are just the tip of the iceberg.

Summary and next steps

In this article we explained the difference between messaging APIs, protocols, and clients. Each approach has its own “sweet spot” and depending on your priorities you’ll now be able to decide which solution is right for you.

To learn more about enterprise messaging and IBM MQ, complete the IBM MQ Developer Essentials Badge.

Understanding enterprise messaging APIs and protocols

Understanding enterprise messaging APIs and protocols