Digital Developer Conference: Hybrid Cloud 2021. On Sep 21, gain free hybrid cloud skills from experts and partners. Register now

Process streams and extract insights with ksqlDB

Summary

This code pattern demonstrates one of the use cases of ksqlDB in an existing event-driven microservices architecture. The code pattern will guide you how to easily add a new feature on to an existing application that leverages the data from the Kafka records.

Description

The previous code pattern, Scaling an Event Driven architecture using an Event Driven autoscaler, walked you through creating a Kafka cluster, creating microservices to interact with it, and deploying an example food ordering application. In this code pattern, you will extend that code pattern and create a ksqlDB database. You will then create a microservice to connect to that ksqlDB database and expose the data in it.

With ksqlDB, you can easily process streams in your application without writing a significant amount of code. The ksqlDB leverages the existing Kafka cluster and you can quickly add new capabilities in your applications. In this code pattern, you will add a new feature in the app that shows the most popular restaurants. The ksqlDB then processes the streams of orders to create that materialized view in real-time. A node.js is used to do push queries and expose that data to the UI in real-time.

When you have completed this code pattern, you will understand how to:

  • Create a ksqlDB and create streams and tables.
  • Add new capabilities in an existing application.
  • Create a microservice to query the ksqlDB.

Flow

Scaling Kafka architecture, adding a ksqldb database

The flow steps 1 – 10 in the architecture are the same as in the previous code pattern, Scaling an Event Driven architecture using an Event Driven autoscaler.

  • 11 – Streams are created from the Kafka topic. A table is also created to get the most popular restaurants.
  • 12 – A Node.js microservice is using the available REST API to do a push query in the “favorite restaurants” table.
  • 13 – The frontend UI is connected to the ksqlDB microservice using websockets. The UI is updated whenever there are new popular restaurants.

Instructions

Find the detailed steps for this pattern in the readme file. The steps will show you how to:

  1. Create and configure the ksqldb database.
  2. Execute KSQL statements.
  3. Build a microservice that exposes data from ksqlDB.
  4. Create ksqlDB credentials, and then deploy the microservice.
  5. Run the application.