Kafka Monthly Digest – February 2021
See what's going on in the Kafka community this month
This is the 37th edition of the Kafka Monthly Digest! In this edition, I’ll cover what happened in the Apache Kafka community in February 2021.
For last month’s digest, see Kafka Monthly Digest: January 2021.
The 2.8.0 minor release reached feature freeze on February 3, and according to the plan the 2.8 branch was cut on February 18. We are now in the stabilization period, and the first release candidate should be available in the next few days. For more details, see the release plan.
The release process of 2.6.2 continued. A few blocker issues in Connect (KAFKA-12326, KAFKA-12339, KAFKA-12340, KAFKA-12343) have delayed the first release candidate but they have now all been fixed. So we’re expecting RC0 in the next few days. All the details are in the release plan.
Last month, the community submitted 6 KIPs (KIP-711 to KIP-716), and these are the ones that caught my eye.
KIP-712: Shallow Mirroring. This KIP is for MirrorMaker1. When a record is mirrored, it is essentially consumed from a cluster and produced to another. This creates a number of overheads as bytes consumed are converted into
ConsumerRecordobjects (and potentially decompressed) before being converted back to bytes (and recompressed) by the producer. This KIP proposes adding a new consumer configuration,
fetch.raw.bytes, to keep the data as
ByteBuffer. The Produce API will also be updated to accept raw bytes as input.
KIP-713: Validation of Enums in configuration. This KIP aims at improving error messages when configurations (client and server side) are not set correctly. The new messages should clearly state the issues and include valid values.
KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2. In order to translate offsets between clusters, MirrorMaker2 uses an offset-syncs topic. When mirroring data from a source cluster to a target cluster, MirrorMaker2 currently creates the offset-syncs topic on the source cluster. This prevents granting only
ReadACLs on the source cluster. This KIP proposes making the location of that topic configurable so it can be moved to the target cluster instead.
In this section, I will cover releases of some community projects. This only includes projects that are Open Source.
- Sarama 1.28.0. Sarama is a pure Golang Kafka client. This new release adds support for Kafka 2.7.0. Consumers can now take advantage of KIP-392 and fetch records from their closest replica. Finally, the
roundrobinpartition assignment strategy has been updated to match the Java client
- Saga Orchestration for Microservices Using the Outbox Pattern
- Real-time monitoring of Formula 1 telemetry data on Kubernetes with Grafana, Apache Kafka, and Strimzi
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.