Stream data with Apache Kafka into the IBM Db2 Event Store

This tutorial is part of the Db2 Event Store learning path.

IBM Db2 Event Store is an in-memory database designed to rapidly ingest and analyze streamed data in event-driven applications. It provides the fabric for fast data with its ability to process massive volume of events in real-time, coupled with optimization for streamed data performance for advanced analytics and actionable insights.

In this tutorial, we’ll run an end-to-end application written on top of the IBM Db2 Event Store. This application is representative of a simplified IoT use-case where sensor data is being streamed to the Event Store and visualized.

Learning objectives

In addition to running the IBM Db2 Event Store, you will also accomplish the following:

• Run a Scala application with sbt
• Create a Kafka Cluster
• Understand the IBM Db2 Event Store Scala API
• Install and use Grafana

Prerequisites

For this tutorial, we utilize Docker, which can be used on a variety of platforms: Windows, MacOS, or Linux. While this tutorial was built on MacOS, it could be easily adjusted to the other platforms supported by the IBM Db2 Event Store Developer Editor, Windows or Linux.

Specifically, the following is required:

• IBM Db2 Event Store Developer Edition 1.1.4
• sbt 0.13.16
• Docker Version 18.06.1-ce-mac73 (26764)
• Enough CPU and memory to run docker, specifically a minimum of 6 CPU and 8 GB.
• Grafana 5.3.2

Estimated time

Completing this tutorial should take about one hour.

Steps

Install the IBM Db2 Event Store Developer Edition

The first step is to install the IBM Db2 Event Store. The Event Store is available as a Desktop Electron App running on top of docker.

• Download the platfom specific and latest installer
• Note: for MacOS, the file is available here:
• MacOS or Windows
• Start the installer (dmg or exe) and accept all defaults
• This operation may take some time based on your bandwith

Install dependencies

Let’s first install the streaming connector for Kafka as well Grafana before putting the application in motion.

Clone the IBM Db2 Event Store streaming connector for Kafka

• Install sbt at the version 0.13.16
  • Optionally add sbt to ‘PATH’ by modifying ‘/etc/paths’ in order to validate that SBT is aligned against the correct 0.13.16 version.

./bin/sbt sbt-version

• Clone the Kakfa Streaming connector for the IBM Db2 Event Store

git clone git@github.com:IBMProjectEventStore/db2eventstore-kafka.git
cd db2eventstore-kafka

Install Grafana

• Install Grafana 5.3.2

brew update 
brew install grafana
brew services restart grafana

• Install the IBM Db2 Event Store data source for Grafana

wget https://github.com/IBMProjectEventStore/db2eventstore-grafana/releases/download/ReactiveSummit2018/db2-event-store-grafana-1.1.tar 
mkdir -p /usr/local/var/lib/grafana/plugins/db2-event-store
mv db2-event-store-grafana-1.1.tar /usr/local/var/lib/grafana/plugins/db2-event-store
cd /usr/local/var/lib/grafana/plugins/db2-event-store
tar -zxvf db2-event-store-grafana-1.1.tar
brew services restart grafana

Compile the IBM Db2 Event Store streaming connector for Kafka

cd db2eventstore-kafka
sbt clean
sbt compile

Initialize the IBM Db2 Event Store developer Edition

• Open the IBM Db2 Event Store Developer Edition
  • Open the Scala notebook “Introduction to IBM Db2 Event Store Scala API”
  • Select the icon “Copy to Notebooks”
  • Run the first 2 cells to create the TESTDB database

Once the Jupyter notebook cell that creates the database has been run, the IBM Db2 Event Store is ready for our sensor.

• Run the REST Server initialization leveraged by the Grafana dashboard

This step will initialize the Event Store for the Grafana Data Source support. It enables the IBM Db2 REST Server to accept external requests.

curl -X POST -H "Content-Type: application/json" -H "authorization: Bearer token" 'http://0.0.0.0:9991/com/ibm/event/api/v1/init/engine?engine=173.19.0.1:1100&rContext=Desktop'

The terminal will return {“code”:”ES100″,”message”:”Engine initialization succeeded”}

The Grafana dashboard

• Login to Grafana
  • Use the default credentials ‘admin’ and ‘admin’
• Add a new Data Source to the IBM Db2 Event Store
• Add a Db2 Event Store Data Source
• Change host port to 9991
• Select ‘Refresh’ and ‘Save & Test’

Add a new Data Source to the IBM Db2 Event Store

Open the provided JSON dashboard db2eventstorevisualization.json and copy its content in your clipboard.

Import the Dashboard by pasting the JSON

Run the sample code

In order to ingest data into the event store, we will first publish batches of JSON payload into the topic of a Kafka cluster. The streaming connector Event Store extension can create its own local Kafka cluster or can be configured against an external one.

In order to run the sample code, we’ll execute two applications:

• The JSON generator dataLoad/run. This generator simulates a minimal sensor driven payload and publishes each batch on a Kakfa topic
• The main streaming connector component eventStream/run. This connector leverages the spark streaming and kakfa integration library to consume Kafka messages and ingest them into the Event Store using a configurable batch size.

• Run the Kakfa Streaming connector for the IBM Db2 Event Store

  • localBroker [Boolean]- true to use of an internal, single node, Kafka broker. False to use an externally configured Broker
  • kafkaBroker [String]- Location of the Kafka broker, pass in “localhost:9092” for a local setup or the ip:port location for an external setup
  • topic [String]- The Kafka topic to be used for the stream
  • eventStore [String]- The IP configuration for the IBM Db2 Event Store
  • database [String]- The Db2 Event Store Database to create or use
  • user [String]- The Db2 Event Store user name to use
  • password [String]- The Db2 Event Store password to use
  • metadata [String]- The type of metadata for this IoT device, for instance “sensor” or “appliance” or “weatherstation”
  • streamingInterval [Long]- The Long value defining the length of the Apache Spark streaming window, in milliseconds
  • batchSize [Int] – The size of the batch to send to the IBM Db2 Event Store

• Run the JSON generator

  • localBroker [Boolean]- true to use of an internal, single node, Kafka broker. False to use an externally configured Broker
  • kafkaBroker [String]- Location of the Kafka broker, pass in “localhost:9092” for a local setup or the ip:port location for an external setup
  • tableName [String] – The table name that will be created within the IBM Db2 Event Store
  • topic [String]- The Kafka topic to be used for the stream
  • group [String]- The Kafka group to be used for the stream
  • metadata [String]- The type of metadata for this simulated IoT device, for instance “sensor” or “appliance” or “car”, …
  • metadataId [Long]- The Long value that can identify this IoT device, for instance “238” or “002” or …
  • batchSize [Int]- The size of the batch that will be sent to the Kafka queue. It can be as small as 1.

sbt "eventStream/run -localBroker true -kafkaBroker localhost:9092 -topic estopic -eventStore localhost:1100 -database TESTDB -user admin -metadata sensor -password password -metadata ReviewTable -streamingInterval 5000 -batchSize 10"
sbt "dataLoad/run -localBroker true -kafkaBroker localhost:9092 -tableName ReviewTable -topic estopic -group group -metadata sensor -metadataId 238 -batchSize 10"

Visualize the data

As the application starts, the grafana dashboard will begin to visualize the incoming sensor data.

The Sensor data

Next steps: learn about the IBM Fast Data Platform

A much more complete tutorial is available on the IBM Db2 Event Store enablement repository, including a full end-to-end application that can be deployed using the IBM Fast Data Platform. Access to this advanced tutorial.

Stop the IBM Db2 Event Store and delete the database

To delete the IBM Db2 Event Store database, shut down the IBM Db2 Event Store Developer Edition and issue the following commands:

docker stop $(docker ps -aq)
docker rm $(docker ps -aq)
cd ~/Library/Application\ Support/ibm-es-desktop
rm -rf zookeeper alluxio ibm

Summary

In this tutorial, we first installed the IBM Db2 Event Store developer Edition. Next, we generated a JSON payload representative of a sensor payload and published it in batches on an Apache Kafka cluster. Once available in Kafka, we used the Apache Spark Streaming and Kafka integration to access batches of payloads and ingest them in the IBM Db2 Event Store. Once in the IBM Db2 Event Store, we connected Grafana to the REST server of the IBM Db2 Event Store in order to run some simple predicates and visualize the results.

References

• The IBM DB2 Event Store
• Db2 Event Store Scala API
• IBM DB2 Event Store Rest API
• Spark Streaming + Kafka Integration Guide
• Watson Studio
• Watson Studio Local
• The IBM Fast Data Platform
• Fast Data
• Apache Spark
• Apache Kafka
• Akka
• Grafana

Conclusion

This tutorial explained how to stream data with Apache Kafka into the Db2 Event Store. The tutorial is the final part of the Learning Path: Db2 Event Store series. Congratulations! You should now have a fundamental understanding of Db2 Event Store and some of its advanced features. But, if you want to learn more about Db2 Event Store, please visit.