Kafka Monthly Digest – March 2021
See what's going on in the Kafka community this month
This is the 38th edition of the Kafka Monthly Digest! In this edition, I’ll cover what happened in the Apache Kafka community in March 2021.
For last month’s digest, see Kafka Monthly Digest: February 2020.
On February 23, Konstantine Karantasis volunteered to run the 3.0.0 major release. This will be the next release after 2.8. As decided by the community, the 3.x series will be able to run Kafka with or without ZooKeeper, and ZooKeeper support will only be completely removed in 4.0.
The next phase for 3.0.0 is KIP freeze. This is currently targeted for early June. See the release plan in the wiki for all the details.
John Roesler published RC0 on March 30 and the vote is ongoing.
Kafka 2.8.0 will introduce the option to run Kafka without ZooKeeper in early access, but this is not for production. This mode is named KRaft, for Kafka Raft. There is currently no migration path and many features are still disabled, but if you want to try it out in a test cluster before 3.0.0, see the KRaft README.
On March 18, I published the first release candidate for 2.7.1. Shortly after, a blocker issue with Kafka Streams was identified (KAFKA-12508). I published a second release candidate, RC1, on March 26. The vote is currently in progress.
The release process of 2.6.2 continued. Sophie Blee-Goldman produced the first release candidate, RC0, on March 12. KAFKA-12508 also impacted this release. A new release candidate should be available in the next few days.
Last month, the community submitted 12 KIPs (KIP-717 to KIP-728), and these are the ones that caught my eye.
These 4 KIPs are in preparation for Kafka 3.0. As 3.0 is a major release, the project will change the default behavior of some features and may make non-backward compatible changes.
KIP-720: Deprecate MirrorMaker v1. MirrorMaker2 has now been available for over a year (Kafka 2.4.0, December 2019), and it aims to fully replace its predecessor. So the community is considering deprecating MirrorMaker 1 for Kafka 3.0 before fully removing it in the next major release.
KIP-722: Enable connector client overrides by default. Kafka Connect client overrides were added in Kafka 2.3. This enables connectors to customize the clients (Consumers, Producers, and Admin) that the Connect runtime will use to grant different identities (and ACLs) to different connectors or to fine tuned these clients for specific connectors. However, this feature is still disabled by default because, in very specific cases, it might cause backward compatibility issues. This KIP proposes changing this setting and enabling all client overrides (set
all) by default.
KIP-724: Drop support for message formats v0 and v1. Kafka uses Message Formats to describe and store records. Since 0.11 (released in 2017), Kafka uses Message Format v2. Today, brokers still support older formats and are able to convert messages if necessary. This proposal aims to remove support for older formats (v0 and v1) in 3.0 to improve broker maintainability as all modern clients use the latest format already.
KIP-726: Make the CooperativeStickyAssignor as the default assignor. Consumers use Assignors to compute the partition assignment of members within consumer groups. Kafka currently provides 4 assignors: Range, RoundRobin, Sticky and CooperativeSticky. CooperativeSticky was introduced in Kafka 2.4 and offers a number of improvements over the current default, Range. This KIP proposes using CooperativeSticky by default.
In this section, I will cover releases of some community projects. This only includes projects that are open source.
- strimzi-kafka-operator 0.22: Strimzi is a Kubernetes Operator for running Kafka. This new version improves the management of Connect clusters and connectors. It also introduces a new API version,
v1beta2, to prepare for the adoption of Kubernetes 1.22.
- Azkarra Streams 0.9: Azkarra is a framework to help build streaming microservices using Kafka Streams. This release allows fine-tuning of RocksDB. The UI dashboard has also been updated to provide a better user experience, and Kafka 2.7 is now supported.
- Capturing Every Change From Shopify’s Sharded Monolith
- Quick profiling of data in Apache Kafka using kafkacat and visidata
- Understanding Non-Key Joins With the Quarkus Extension for Kafka Streams
- Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.