For Internet of Things (IoT) devices, connecting to the Internet is kind of a requirement. Connecting to the Internet allows the devices to work with each other and with backend services. The underlying network protocol of the Internet is TCP/IP. Built on top of the TCP/IP stack, MQTT (Message Queue Telemetry Transport) has become the standard for IoT communications. MQTT can also run on SSL/TLS, which is a secure protocol built on TCP/IP, to ensure that all data communication between devices are encrypted and secure.
MQTT was originally invented and developed by IBM in the late 1990’s. Its original application was to link sensors on oil pipelines with satellites. As its name suggests, it is a messaging protocol that supports asynchronous communication between parties. An asynchronous messaging protocol de-couples the message sender and receiver in both space and time, and hence is scalable in unreliable network environments.
Despite its name, it actually has nothing to do with messaging queues, and uses a publish and subscribe model instead. In late 2014, it officially became an OASIS open standard, and it is supported in popular programming languages by using multiple open source implementations.
What is MQTT and why should I use it?
MQTT is a lightweight and flexible network protocol that strikes the right balance for IoT developers:
- The lightweight protocol allows it to be implemented on both heavily constrained device hardware as well as high latency / limited bandwidth networks.
- Its flexibility makes it possible to support diverse application scenarios for IoT devices and services.
To understand why MQTT is such a good fit for IoT developers, let’s first examine why other popular network protocols failed in IoT.
Why not … the rest of the alphabet soup of network protocols
Most developers are already familiar with HTTP web services. So, why not just have IoT devices connect to web services? The device could send in its data as a HTTP request and receive updates from the system as the HTTP response. This request and response pattern does have some severe limitations:
- HTTP is a synchronous protocol. The client waits for the server to respond. That is a requirement for web browsers, but it comes at the cost of poor scalability. In the world of IoT, the large number of devices and most likely an unreliable / high latency network have made synchronous communication problematic. An asynchronous messaging protocol is much more suitable for IoT applications. The sensors can send in readings, and let the network figure out the optimal path and timing for delivery to its destination devices and services.
- HTTP is one-way. The client must initiate the connection. In an IoT application, the devices or sensors are typically clients, which means that they cannot passively receive commands from the network.
- HTTP is a 1-1 protocol. The client makes a request, and the server responds. It is difficult and expensive to broadcast a message to all devices on the network, which is a common use case in IoT applications.
- HTTP is a heavy weight protocol with many headers and rules. It is not suitable for constrained networks.
For the above reasons, most high performance scalable systems use an asynchronous messaging bus, rather than web services, for internal data exchange. In fact, the most popular messaging protocol used in enterprise middleware systems is called AMQP (Advanced Message Queuing Protocol). However, in the high-performance environment, computing power and network latency are typically not a concern. AMQP is designed for reliability and interoperability in enterprise applications. It has a rich feature set, but it is not suitable for resource-constrained IoT applications.
Besides AMQP, there are other popular messaging protocols. For example, the XMPP (Extensible Messaging and Presence Protocol) is a peer-to-peer instant messaging (IM) protocol. It is heavy on features that support IM use cases, such as presence and media attachments. Compared with MQTT, it requires much more resources both on the device and on the network.
So, what makes the MQTT so lightweight and flexible? A key feature of the MQTT protocol is its publish and subscribe model. As with all messaging protocols, it decouples the publisher and consumer of data.
The publish and subscribe model
The MQTT protocol defines two types of entities in the network: a message broker and a number of clients. The broker is a server that receives all messages from the clients and then routes those messages to relevant destination clients. A client is anything that can interact with the broker to send and receive messages. A client could be an IoT sensor in the field or an application in a data center that processes IoT data.
- The client connects to the broker. It can subscribe to any message “topic” in the broker. This connection can be a plain TCP/IP connection or an encrypted TLS connection for sensitive messages.
- The client publishes messages under a topic by sending the message and topic to the broker.
- The broker then forwards the message to all clients that subscribe to that topic.
Since MQTT messages are organized by topics, the application developer has the flexibility to specify that certain clients can only interact with certain messages. For example, sensors will publish their readings under the “sensor_data” topic, and subscribe to the “config_change” topic. Data processing applications that save sensor data into a backend database will subscribe to the “sensor_data” topic. An admin console application could receive system admin’s commands to adjust the sensors’ configurations, such as sensitivity and sample frequency, and publish those changes to the “config_change” topic.
Figure 1. The MQTT publish and subscribe model for IoT sensors
At the same time, MQTT is lightweight. It has a simple header to specify the message type, a text-based topic, and then an arbitrary binary payload. The application can use any data format for the payload, such as JSON, XML, encrypted binary, or Base64, as long as the destination clients can parse the payload.
Getting started with MQTT development
The easiest tool to get started with MQTT development is the mosquitto module, which is part of the Eclipse project that provides MQTT SDKs and libraries in multiple programming languages. It contains an MQTT broker that can run on your local computer and command line tools to interact with the broker by using messages.
You can download and install the module from the mosquitto web site. Make sure that you install both the mosquitto (the broker) and the mosquitto_clients (the clients) modules.
The install command on Ubuntu Linux is as follows.
$ sudo apt-get install mosquitto mosquitto-clients
The install command on RedHat Linux is as follows.
$ sudo yum -y install epel-release
$ sudo yum -y install mosquitto
The mosquitto command runs the MQTT broker on your local computer. You could optionally use the -d option to run it in the background.
$ mosquitto -d
Alternatively, you could run it as a system service.
$ systemctl start mosquitto
Next, in another terminal window, you can use the mosquitto_sub command to connect to the local broker and subscribe to a topic. Once the command executes, it will wait and print out any message it receives via the subscription as they come in.
$ mosquitto_sub -t "dw/demo"
In yet another terminal window, you can use the mosquitto_pub command to connect to the local broker and then publish a message to a topic.
$ mosquitto_pub -t "dw/demo" -m "hello world!"
Now, the terminal that runs mosquitto_sub should now print out hello world! on the screen. You have just sent and received a message via an MQTT broker!
For more options on how to use the mosquitto tools, as well as on how to use the mosquitto API to create your own MQTT client applications, please refer to its documentation.
Understanding the MQTT protocol
MQTT is a wire protocol that specifies how data bytes should be organized and transmitted over the TCP/IP network. But for practical purposes, developers do not need to understand the wire protocol. All we need to know is that each message has a command and data payload. The command defines the message type (for example, a CONNECT message or a SUBSCRIBE message). All MQTT libraries and tools provide simple ways to manipulate those messages directly and can automatically populate some required fields, such as message and client IDs.
First, the client connects to the broker by sending a CONNECT message. The CONNECT message asks to establish a connection from the client to the broker. The CONNECT message has the following content parameters.
Table 1. CONNECT message parameters
|cleanSession||This flag specifies whether the connection is persistent or not. A persistent session stores all the subscriptions and potentially missed messages (depending on QoS) in the broker. (I discuss QoS later.)|
|username||The broker’s authentication and authorization credentials.|
|password||The broker’s authentication and authorization credentials.|
|lastWillTopic||When the connection is dropped unexpectedly, the broker will automatically publish a “last will” message to a topic.|
|lastWillQos||The “last will” message’s QoS. (I discuss QoS later.)|
|lastWillMessage||The “last will” message itself.|
|keepAlive||This is the time interval that the client needs to ping the broker to keep the connection alive.|
The client will receive a CONNACK message from the broker. The CONNACK message has the following content parameters.
Table 2. CONNACK message parameters
|sessionPresent||This indicates whether the connection already has a persistent session. That is, the connection already has subscribed topics, and will receive the delivery of missed messages.|
|returnCode||0 indicates success. Other values identify the cause of failure.|
After a connection is established, the client can then send one or more SUBSCRIBE message to the broker to indicate that it will receive messages from the broker for certain topics. The message can have one or multiple repeats of the following parameters.
Table 3. SUBSCRIBE message parameters
|qos|| The qos (quality of service, or QoS) flag indicates how consistently the messages under this topic needs to be delivered to clients. |
|topic||A topic to subscribe to. A topic can have multiple levels separated by the slash character. For example, “dw/demo” and “ibm/cloud/mqtt” are valid topics.|
After a client has successfully subscribed to a topic, the broker returns a SUBACK message with one or more returnCode parameters.
Table 4. SUBACK message parameters
|returnCode|| A return code exists for each of the topics in the SUBCRIBE command. The return values are as follows. |
Corresponding to the SUBSCRIBE message, the client can also UNSUBSCRIBE from a topic or multiple topics.
Table 5. UNSUBSCRIBE message parameters
|topic||This parameter can repeat for multiple topics.|
The client can send PUBLISH messages to the broker. The message contains a topic and data payload. The broker then forwards the message to all clients that subscribe to that topic.
Table 6. PUBLISH message parameters
|topicName||The topic under which the message is published.|
|qos|| The quality of service level of the message delivery. (See |
|retainFlag||This flag indicates whether the broker will retain the message as the last known message for this topic.|
|payload||The actual data in the message. It can be a text string or a binary blob of data.|
Tips and workarounds
The power of MQTT is its simplicity. You have no constraints on the kind of topics or message payloads that you can use. That allows for some interesting use cases. For example, consider these questions:
How do I do 1-1 messages with MQTT? Both parties can agree to use a topic that is unique to them. For example, the topic name could contain the IDs of both clients to ensure its uniqueness.
How does a client broadcast its presence status? The system could have an agreed upon naming convention for “presence” topics. For example, the “presence/client-id” topic could have the presence information for a client. The client sets the message to true when it connects and false when it disconnects. The client can also set a last will message to false so that it is set when the connection drops. The message can be retained by the broker so that new clients can read the topic and find out the presence status.
How do I secure the communications? The client-to-broker connection can be an encrypted TLS connection to protect the data in transit. In addition, since the MQTT protocol imposes no constraints on the payload data format, the system could have an agreed upon encryption method and key update mechanism. After that, all content in the payload could be encrypted binary data of the actual JSON or XML messages.
In this article, I gave a technical introduction to the MQTT protocol. You learned what is MQTT, what makes MQTT suitable for IoT applications, and how to get started with developing apps that use MQTT.