Stream and store retail order data for analysis
Use Akka to implement a WebSockets endpoint and stream data to IBM Db2 Event Store
This code pattern is part of the Db2 Event Store learning path.
|100||Achieve real time analytics, IoT, and fast data to gather meaningful insights||Blog|
|101||Understand customer interests with clickstream analysis||Code pattern|
|102||Analyze IoT sensor data with machine learning and advanced analytics||Code pattern|
|103||Stream and store retail order data for analysis||Code pattern|
|104||Stream data with Apache Kafka into the IBM Db2 Event Store||Tutorial|
In this code pattern, we’ll build a Scala app that uses Akka to implement a WebSockets endpoint which streams data to a Db2 Event Store database. For our data, we’ll use online retail order details in CSV format. We’ll use Jupyter notebooks with Scala and Brunel to visualize the Event Store data.
In order to quickly react to changes in your business, you need to have high-speed data collection for current event-driven and also historical analysis. We’re using thin, fast WebSockets clients to send messages via WebSockets and letting the Akka toolkit handle the message streams, transform the data, and feed it into Event Store.
Akka HTTP gives us a very easy way to provide the WebSockets endpoint. Akka Streams is a powerful and elegant way to move the data from source to sink. We’re using Akka Streams with Alpakka to parse and transform the data from CSV text strings or CSV files into Spark SQL Rows. We use a simple
divertTo function to decide which messages are new orders and which ones are cancellations.
We’re providing code to implement EventStoreSink and EventStoreFlow. This ultimately pushes the transformed data into Event Store tables. Once the data is in Event Store, we use the bundled Jupyter notebooks environment to analyze and visualize the data.
This is a simple example you can run on a laptop, but it’s built with tools that are designed to scale. It can easily be extended, whether you want to build a client, use more Akka functionality to handle the data streams, or flex your data science muscles with Event Store and Jupyter notebooks.
After completing this code pattern, you’ll understand how to:
- Implement WebSockets with Scala and Akka.
- Use Alpakka Flows for CSV parsing and inserts into Event Store
- Use Jupyter notebooks and Scala to interact with Event Store
- Use Spark SQL and Brunel visualizations to analyze the data
- Setup the database with a Jupyter Notebook.
- Submit CSV data via WebSockets.
- Use Akka and Alpakka to transform the data and feed it into Event Store.
- Present the data with Brunel visualizations in a Jupyter Notebook.
Get the detailed instructions in the README file. These steps will show you how to:
- Clone the repo.
- Install IBM Db2 Event Store Developer Edition.
- Run the database setup notebook.
- Run the Scala app.
- Feed in data.
- Visualize the data.
This code pattern showed you how to use Akka to implement a WebSockets endpoint and stream data to IBM Db2 Event Store. The code pattern is part of the Learning Path: Db2 Event Store Learning path series. To continue the series and learn about Db2 Event Store features, take a look at the next code tutorial, Stream data with Apache Kafka into the IBM Db2 Event Store.