Kafka Monthly Digest – January 2021
From releases, to Kafka Improvement Proposals (KIPs), to blogs, check out what's going on in the Apache Kafka community for January 2021.
This is the 36th edition of the Kafka Monthly Digest! This series has now been running for 3 full years! Thanks for all the great feedback. Let’s keep going! If you are interested in a bit of history about this series, I’ve shared a few details on Twitter.
In this edition, I’ll cover what happened in the Apache Kafka community in January 2021.
For last month’s digest, see Kafka Monthly Digest: December 2020.
On January 7, John Roesler volunteered to run the 2.8.0 release. KIP freeze happened on January 27. The next step will be feature freeze on February 3. As always, the release plan is available on the wiki.
A regression in Kafka Streams was discovered in Kafka 2.6.1. This issue, KAFKA-12190, causes crashes for users running on Windows. Sophie Blee-Goldman volunteered to run this new bugfix release. All the details are in the release plan.
Last month, the community submitted 9 KIPs (KIP-702 to KIP-710), and these are the ones that caught my eye.
KIP-707: The future of KafkaFuture. The Admin API makes extensive use of
KafkaFutureto provide sync and async functionalities. This class was designed when Java 7 was still supported. With Java 8,
CompletableFuturebecame the new preferred API as it’s richer and many 3rd party tools or APIs support it directly. This KIP proposes extending
KafkaFutureto be able to retrieve
KIP-708: Rack aware Kafka Streams with pluggable StandbyTask assignor. Kafka Streams applications can set up standby tasks to provide fast recovery times in case a task fails. This KIP aims at improving this mechanism even further by automatically distributing standby tasks across racks if possible.
KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters. MirrorMaker 2 runs in Kafka Connect. However, when it’s run in dedicated mode, it does not expose Kafka Connect’s REST API. The REST API is key for administrators to efficiently monitor and manage Kafka Connect deployments, but it’s also used by Connectors to dynamically react to topic and group changes. The goal of this KIP is to address these issues and expose all Kafka Connect capabilities when MirrorMaker 2 is run in dedicated mode.
In this section, I will cover releases of some community projects. This only includes projects that are Open Source.
- Librdkafka 1.6.0. Librdkafka is a Kafka client in C/C++. This new version adds support for incremental rebalancing for consumers. The Admin client now has APIs to delete records, consumer groups and consumer group offsets. Finally, the producer can now use sticky partitioning to achieve higher throughput.
- Debezium 1.4.0. Debezium is a popular Change Data Capture platform. 1.4.0 includes a brand new connector for Vitess and also includes improvements to the Oracle, PostgreSQL, and MySQL connectors.
- strimzi-kafka-operator 0.21: Strimzi is a Kubernetes Operator for running Kafka. Kafka 2.7.0 is now supported, managing the configuration of Connectors has been simplified and overall most components have received improvements.
- Implementing mTLS and Securing Apache Kafka at Zendesk
- Debezium in 2020 — The Recap!
- Optimizing Kafka consumers
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.