Apache Kafka is a distributed streaming platform. It is made up of the following components: Broker, Producer, Consumer, Admin, Connect, and Streams.

Architecture diagram for Apache Kafka components

Learning objectives

In this workshop, you'll learn how to build an end-to-end streaming pipeline using Apache Kafka, Kafka Connect, and Kafka Streams.

Architecture diagram of the end-to-end streaming pipeline

You'll learn how to:

Configure the Kafka command line tools
Create, list, and describe topics using the kafka-topics.sh tool
Consume records with the kafka-console-consumer.sh tool
Produce records with the kafka-console-producer.sh tool
Describe consumer groups with the kafka-consumer-group.sh tool
Configure and run the Kafka Connect runtime in distributed mode
Configure and run the FileStreamsSourceConnector Kafka connector
Run a Kafka Streams application

Prerequisites

Apache Kafka CLI
Java SDK, Version 8 or above
gradle, Version 6 or above

Estimated time

Completing this workshop should take about 1 hour.

Step 1: Install and configure a Kafka cluster

In part 1 of this workshop, you set up a Kafka cluster:

Step 2: Sending and consuming messages

In part 2 of this workshop, you'll use the Kafka CLI to create a topic, send some messages, and consume some messages. You'll also learn how to set up a consumer group.

Step 3: Integrating data with Kafka Connect

In part 3 of this workshop, you configure the Kafka Connect runtime for your environment:

For IBM Event Streams on IBM Cloud
For a local Kafka cluster

Then, you configure, start, and test the connector.

Step 4: Processing data with Kafka Streams

In part 4 of this workshop, you learn how to use Kafka Streams to process streams of data in realtime using the built-in sample application.

Summary

In this workshop, you learned how to build an end-to-end streaming pipeline, with data flowing into Apache Kafka from an external system and then doing realtime processing on that data. By exploring the key concepts and components of Apache Kafka in this workshop, you can now build reliable, scalable, and performant environments.

Next steps

Perhaps you're ready to try one of these tutorials and code patterns to build your Kafka skills further:

Workshop: Get started with Apache Kafka

Workshop: Get started with Apache Kafka