Workshop: Get started with Apache Kafka

Apache Kafka is a distributed streaming platform. It is made up of the following components: Broker, Producer, Consumer, Admin, Connect, and Streams.

Architecture diagram for Apache Kafka components

Learning objectives

In this workshop, you’ll learn how to build an end-to-end streaming pipeline using Apache Kafka, Kafka Connect, and Kafka Streams.

Architecture diagram of the end-to-end streaming pipeline

You’ll learn how to:

  • Configure the Kafka command line tools
  • Create, list, and describe topics using the kafka-topics.sh tool
  • Consume records with the kafka-console-consumer.sh tool
  • Produce records with the kafka-console-producer.sh tool
  • Describe consumer groups with the kafka-consumer-group.sh tool
  • Configure and run the Kafka Connect runtime in distributed mode
  • Configure and run the FileStreamsSourceConnector Kafka connector
  • Run a Kafka Streams application

Prerequisites

Estimated time

Completing this workshop should take about 1 hour.

Steps

  1. Install & configure a Kafka cluster
  2. Sending & consuming messages
  3. Integrating data with Kafka Connect
  4. Processing data with Kafka Streams

Step 1: Install and configure a Kafka cluster

In part 1 of this workshop, you set up a Kafka cluster:

Step 2: Sending and consuming messages

In part 2 of this workshop, you’ll use the Kafka CLI to create a topic, send some messages, and consume some messages. You’ll also learn how to set up a consumer group.

Step 3: Integrating data with Kafka Connect

In part 3 of this workshop, you configure the Kafka Connect runtime for your environment:

Then, you configure, start, and test the connector.

Step 4: Processing data with Kafka Streams

In part 4 of this workshop, you learn how to use Kafka Streams to process streams of data in realtime using the built-in sample application.

Summary

In this workshop, you learned how to build an end-to-end streaming pipeline, with data flowing into Apache Kafka from an external system and then doing realtime processing on that data. By exploring the key concepts and components of Apache Kafka in this workshop, you can now build reliable, scalable, and performant environments.

Next steps

Perhaps you’re ready to try one of these tutorials and code patterns to build your Kafka skills further: