Kafka Monthly Digest – April 2021
See what's going on in the Kafka community this month
This is the 39th edition of the Kafka Monthly Digest! In this edition, I’ll cover what happened in the Apache Kafka community in April 2021.
For last month’s digest, see Kafka Monthly Digest: March 2020.
Kafka 2.8.0 introduces the option to run Kafka without ZooKeeper in early access (KIP-500). This mode is named KRaft, for Kafka Raft. This is not for production as there is currently no migration path and many features are still disabled, but if you want to try it out in a test cluster before 3.0.0, see the KRaft README.
This new minor version brings a number of interesting features:
- The computed configuration of Connector tasks can now be retrieved via the REST API. (KIP-661)
I published the third release candidate, 2.7.1-RC2, on April 8. The vote passed on May 6 and I’ll release 2.7.1 in the next few days.
Kafka 3.0.0 is the next major release. As mentioned last month, the next phase is KIP freeze. This is currently targeted for early June. See the release plan in the wiki for all the details.
Last month, the community submitted 9 KIPs (KIP-729 to KIP-737), and these are the ones that caught my eye.
KIP-730: Producer ID generation in KRaft mode. Until now, producer IDs generated for idempotent and transactional producers were stored in ZooKeeper. With KIP-500 and ZooKeeper going away, this KIP proposes a new mechanism to generate and manage producer IDs. A new API,
AllocateProducerIds, will be used by brokers to request producer IDs from the controller.
KIP-731: Record Rate Limiting for Kafka Connect. This KIP’s goal is to provide a quota mechanism to Kafka Connect. This will allow administrators to define a maximum record rate, consume rate and produce rate per task per connector. This will useful when running multiple connectors in a Kafka Connect cluster to avoid them fighting for resources. This could also be used to prevent Connect from overloading one of the systems it connects to, including Kafka!
KIP-734: Improve AdminClient.listOffsets to return timestamp and offset for the record with the largest timestamp. This KIP proposes extending
listOffsets()to also optionally return the largest timestamp alongside the offset. The largest timestamp can be good indication of “liveness” for a partition.
In this section, I will cover releases of some community projects. This only includes projects that are open source.
- Debezium 1.5.0: As usual there are a lot of new features and fixes in this new release of Debezium. Most notably, there’s a brand new implementation of the MySQL connector (with transaction metadata support) and the LogMiner-based Oracle connector is now marked as stable.
The Kafka Summit Europe is on May 11 and 12. It is a virtual event with over 100 sessions covering all aspects of the Kafka ecosystem. You can register for free on the Kafka Summit website.
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.