We’re giving away 1,500 DJI Tello drones. Enter to win ›
By Michael Yuan | Published May 12, 2017 - Updated August 7, 2017
For Internet of Things (IoT) devices, connecting to the internet is a requirement. Connecting to the internet allows the devices to work with each other and with backend services. The underlying network protocol of the internet is TCP/IP. Built on top of the TCP/IP stack, MQTT (Message Queue Telemetry Transport) has become the standard for IoT communications.
MQTT was originally invented and developed by IBM in the late 1990’s. Its original application was to link sensors on oil pipelines with satellites. As its name suggests, it is a messaging protocol that supports asynchronous communication between parties. An asynchronous messaging protocol de-couples the message sender and receiver in both space and time, and hence is scalable in unreliable network environments. Despite its name, it has nothing to do with messaging queues, and uses a publish and subscribe model instead. In late 2014, it officially became an OASIS open standard, and it is supported in popular programming languages by using multiple open source implementations.
MQTT is a lightweight and flexible network protocol that strikes the right balance for IoT developers:
To understand why MQTT is such a good fit for IoT developers, let’s first examine why other popular network protocols failed in IoT.
Most developers are already familiar with HTTP web services. So, why not have IoT devices connect to web services? The device could send in its data as an HTTP request and receive updates from the system as the HTTP response. This request and response pattern does have some severe limitations:
For the above reasons, most high-performance scalable systems use an asynchronous messaging bus, rather than web services, for internal data exchange. In fact, the most popular messaging protocol that is used in enterprise middleware systems is called AMQP (Advanced Message Queuing Protocol). However, in the high-performance environment, computing power and network latency are typically not a concern. AMQP is designed for reliability and interoperability in enterprise applications. It has a rich feature set, but it is not suitable for resource-constrained IoT applications.
Besides AMQP, there are other popular messaging protocols. For example, the XMPP (Extensible Messaging and Presence Protocol) is a peer-to-peer instant messaging (IM) protocol. It is heavy on features that support IM use cases, such as presence and media attachments. Compared with MQTT, it requires much more resources both on the device and on the network.
So, what makes the MQTT so lightweight and flexible? A key feature of the MQTT protocol is its publish and subscribe model. As with all messaging protocols, it decouples the publisher and consumer of data.
The MQTT protocol defines two types of entities in the network: a message broker and a number of clients. The broker is a server that receives all messages from the clients and then routes those messages to relevant destination clients. A client is anything that can interact with the broker to send and receive messages. A client can be an IoT sensor in the field or an application in a data center that processes IoT data.
Since MQTT messages are organized by topics, the application developer has the flexibility to specify that certain clients can only interact with certain messages. For example, sensors will publish their readings under the “sensor_data” topic, and subscribe to the “config_change” topic. Data processing applications that save sensor data into a backend database will subscribe to the “sensor_data” topic. An admin console application could receive system admin’s commands to adjust the sensors’ configurations, such as sensitivity and sample frequency, and publish those changes to the “config_change” topic. (See Figure 1.)
At the same time, MQTT is lightweight. It has a simple header to specify the message type, a text-based topic, and then an arbitrary binary payload. The application can use any data format for the payload, such as JSON, XML, encrypted binary, or Base64, as long as the destination clients can parse the payload.
The easiest tool to get started with MQTT development is the Python mosquitto module, which is part of the Eclipse Paho project that provides MQTT SDKs and libraries in multiple programming languages. It contains an MQTT broker that can run on your local computer and command line tools to interact with the broker by using messages. You can download and install the mosquitto module from the mosquitto website.
The mosquitto command runs the MQTT broker on your local computer. You can optionally use the -d option to run it in the background.
$ mosquitto -d
Next, in another terminal window, you can use the mosquitto_sub command to connect to the local broker and subscribe to a topic. After the command runs, it will wait and print out any message it receives from the subscription as they come in.
$ mosquitto_sub -t "dw/demo"
In yet another terminal window, you can use the mosquitto_pub command to connect to the local broker and then publish a message to a topic.
$ mosquitto_pub -t "dw/demo" -m "hello world!"
Now, the terminal that runs mosquitto_sub should now print out “hello world!” on the screen. You just sent and received a message by using an MQTT broker!
Of course, in a production system, you cannot use a local computer as your broker. Instead, you might use the IBM Cloud Internet of Things Platform service, which is a reliable and on-demand service that functions like an MQTT broker. (You can read more about how this IBM Cloud service integrates and uses MQTT as its protocol for communicating with devices and applications in the service documentation.)
The IBM Cloud Internet of Things Platform service works as follows.
For each device client, IBM Cloud will assign a host name, user name, and password to connect to your service instance (the MQTT broker). (On IBM Cloud, the user name is always use-token-auth, and the password is the token that is shown in Figure 2 for each connected device.)
When using a remote MQTT broker, you will need to pass in the broker’s host name and authentication credentials to your mosquitto_sub and mosquitto_pub commands. For example, the following command subscribes to the demo topic on our Internet of Things Platform service with the user name and password that is provided by IBM Cloud:
$ mosquitto_sub -t "demo" -h host.iotp.mqtt.bluemix.com -u username -P password
For more options on how to use the mosquitto tools and also on how to use the mosquitto API to create your own MQTT client applications, see the documentation on the mosquitto website.
Now that you have the necessary tools, let’s dive deeper into the MQTT protocol.
MQTT is a wire protocol that specifies how data bytes are organized and transmitted over the TCP/IP network. But for practical purposes, developers do not need to understand the wire protocol. All we need to know is that each message has a command and data payload. The command defines the message type (for example, a CONNECT message or a SUBSCRIBE message). All MQTT libraries and tools provide simple ways to manipulate those messages directly and can automatically populate some required fields, such as message and client IDs.
First, the client connects to the broker by sending a CONNECT message. The CONNECT message asks to establish a connection from the client to the broker. The CONNECT message has the following content parameters.
The client will receive a CONNACK message from the broker. The CONNACK message has the following content parameters.
After a connection is established, the client can then send one or more SUBSCRIBE messages to the broker to indicate that it will receive messages from the broker for certain topics. The message can have one or multiple repeats of the following parameters.
After a client has successfully subscribed to a topic, the broker returns a SUBACK message with one or more returnCode parameters.
Corresponding to the SUBSCRIBE message, the client can also UNSUBSCRIBE from a topic or multiple topics.
The client can send PUBLISH messages to the broker. The message contains a topic and data payload. The broker then forwards the message to all clients that subscribe to that topic.
The power of MQTT is its simplicity. You have no constraints on the kind of topics or message payloads that you can use. That allows for some interesting use cases. For example, consider these questions:
How do I do 1-1 messages with MQTT? Both parties can agree to use a topic that is unique to them. For example, the topic name could contain the IDs of both clients to ensure its uniqueness.
How does a client broadcast its presence status? The system could have an agreed upon naming convention for “presence” topics. For example, the “presence/client-id” topic could have the presence information for a client. The client sets the message to true when it connects and false when it disconnects. The client can also set a last will message to false so that it is set when the connection drops. The message can be retained by the broker so that new clients can read the topic and find out the presence status.
How do I secure the communications? The client-to-broker connection can be an encrypted TLS connection to protect the data in transit. In addition, since the MQTT protocol imposes no constraints on the payload data format, the system could have an agreed upon encryption method and key update mechanism. After that, all content in the payload could be encrypted binary data of the actual JSON or XML messages.
In this article, I gave a technical introduction to the MQTT protocol. You learned what is MQTT, what makes MQTT suitable for IoT applications, and how to get started with developing apps that use MQTT.
In one of my next articles, I show you how to develop a complete IoT sensor solution with back-end MQTT services and using a NodeMCU board.
December 3, 2018
Back to top