Detect anomolies in streaming data

Summary

In the Internet of Things (IoT) world, the sensors send events periodically. This event data is stored, and the data scientist builds machine learning models for actionable insights on the data. The machine learning model can be a prediction model for the equipment parameter going outside of a certain threshhold.

The following scenario is considered for this developer code pattern: In a chemical research plant, the containers with various chemicals under study are required to be maintained within a certain threshold. In our case, the minimum temperature threshold is 27°F, and the maximum is 30°F. If container temperatures are too low or too high, the consequences could be fatal. Hence, swift action must be taken when the container temperature crosses the defined threshold.

Description

This code pattern will cover anomoly detection in IoT event streams using IBM Event Streams and IBM Cloud Pak for Data with a SingleStore database. It will also cover prediction of the container temperatures for future days and detect which day would the container cross the threshold.

Once you complete the code pattern, you will be able to:

  • Send IoT events to IBM Event Streams.
  • Create SingleStore Kafka pipelines to consume event data from IBM Event Streams and store the data in a SingleStore table inside IBM Cloud Pak for Data.
  • Create a machine learning model to predict temperature events on IBM Cloud Pak for Data.
  • Create a Cognos dashboard to visualize the insights on IBM Cloud Pak for Data.

Flow

flow

  1. IoT temperature sensors send data to IBM Event Streams.
  2. Kafka pipelines in SingleStore consume data from IBM Event Streams and store data into tables.
  3. The AI model takes the sensor data from SingleStore at regular intervals as input. The model forecasts the sensor data, predicts anomalies, and stores back the anomaly data to SingleStore.
  4. A dashboard is displayed by Cognos using the AI model output in the SingleStore database.

Instructions

Ready to get started? Check out the README for detailed instructions that explain how to:

  1. Create an instance of IBM Event Streams
  2. Note broker URLs
  3. Create credentials
  4. Download the SSL certificate from Event Streams
  5. Upload the certificate to the SingleStore cluster
  6. Create a database, table, and pipeline
  7. Clone the repo
  8. Build and run the application
  9. Deploy and run the AI model on IBM Cloud Pak for Data
  10. Create dashboard using IBM Cognos Analytics on IBM Cloud Pak for Data