Apache Kafka is a distributed streaming platform. It is made up of the following components: Broker, Producer, Consumer, Admin, Connect, and Streams.
In this workshop, you’ll learn how to build an end-to-end streaming pipeline using Apache Kafka, Kafka Connect, and Kafka Streams.
You’ll learn how to:
- Configure the Kafka command line tools
- Create, list, and describe topics using the
- Consume records with the
- Produce records with the
- Describe consumer groups with the
- Configure and run the Kafka Connect runtime in distributed mode
- Configure and run the FileStreamsSourceConnector Kafka connector
- Run a Kafka Streams application
Completing this workshop should take about 1 hour.
- Install & configure a Kafka cluster
- Sending & consuming messages
- Integrating data with Kafka Connect
- Processing data with Kafka Streams
Step 1: Install and configure a Kafka cluster
In part 1 of this workshop, you set up a Kafka cluster:
Step 2: Sending and consuming messages
In part 2 of this workshop, you’ll use the Kafka CLI to create a topic, send some messages, and consume some messages. You’ll also learn how to set up a consumer group.
Step 3: Integrating data with Kafka Connect
In part 3 of this workshop, you configure the Kafka Connect runtime for your environment:
Then, you configure, start, and test the connector.
Step 4: Processing data with Kafka Streams
In part 4 of this workshop, you learn how to use Kafka Streams to process streams of data in realtime using the built-in sample application.
In this workshop, you learned how to build an end-to-end streaming pipeline, with data flowing into Apache Kafka from an external system and then doing realtime processing on that data. By exploring the key concepts and components of Apache Kafka in this workshop, you can now build reliable, scalable, and performant environments.
Perhaps you’re ready to try one of these tutorials and code patterns to build your Kafka skills further:
- Tutorial: Develop Java programs to produce and consume messages to and from Apache Kafka using the Kafka Producer and Consumer APIs
- Tutorial: Developing a stream processor with Apache Kafka using Kafka Streams
- Code pattern: Determine trending topics with clickstream analysis using Apache Spark and Apache Kafka