Use MQTT to stream real-time data

Summary

MQTT is a real-time publish subscribe protocol that’s well suited for efficient distribution of data. Using a series of microservices, you can convert existing open data about the NY State power grid into an open, real-time streaming service.

Description

The ny-power project uses a set of microservices to consume open data about the performance of the NY State power grid, and computes an approximate level of carbon intensity. This computed data is served in a real-time MQTT stream, making it easy to consume in other applications. It also shows how to manage a set of data processing in Kubernetes with Helm.

When you have completed this code pattern, you will understand how to:

• Use MQTT in applications where streaming access to data is needed
• Use Helm to deploy a Kubernetes application
• Host a non-cloud-native service, like MQTT, in Kubernetes
• Use Kubernetes microservices to process data in a loosely coupled way

Flow

1. The user accesses the website.
2. The web browser directly accesses the MQTT service.
3. The application’s data pump polls NY ISO every minute looking for new data.
4. Data is pushed to MQTT service.
5. The archiver is subscribed to the MQTT service and sends all new data to Influx DB.
6. On any new data it computes current CO2 / kWh, and publishes both it and the recent time series to the MQTT service.