Kafka Monthly Digest – December 2019 and 2019 in review
From releases, to Kafka Improvement Proposals (KIPs), to blogs, check out what's going on in the Apache Kafka community for December 2019.
In this 23rd edition of the Kafka Monthly Digest, I will cover what happened in the Apache Kafka community in December as well as some of the milestones the project and community reached in 2019!
For last month’s digest, see Kafka Monthly Digest: November 2019.
After four Release Candidates, Manikumar Reddy released Apache Kafka 2.4.0 on December 16. This new minor version brings a number of interesting features:
- New Java authorizer Interface. This new interface provides a richer context to authorizers and also supports batching (KIP-504)
- Administrative API for replica reassignment. Reassignments can now be triggered via the AdminClient and are now incremental and cancelable (KIP-455)
- Support for incremental cooperative rebalancing to the consumer rebalance protocol. This new incremental protocol provides shorter rebalances for heavy, stateful consumer applications such as Kafka Streams (KIP-429)
- Allow consumers to fetch from closest replica. Consumers can now fetch from follower replicas to avoid cross datacenter traffic if possible (KIP-392)
- Sticky Partitioner. This new partitioner is optimized for message batching and can significantly improve performance for some workloads (KIP-480)
- ZooKeeper 3.5 support
- Scala 2.13 support
- MirrorMaker 2.0. This new tool allows to mirror multiple clusters and create complex replication topologies. It also supports dynamic topic creation/updates and offsets translation (KIP-382)
- Securing internal Connect REST endpoints. Authentication and Authorizations can be configured to secure the Connect REST API (KIP-507)
- Extend Connect Converter to support headers (KIP-440)
- Support for non-key joining in KTable. A foreign key can now be used for table to table joins (KIP-213)
- Allow to define custom processor names with KStreams DSL (KIP-307)
- TopologyTestDriver test input and output usability improvements (KIP-470)
Last month, the community submitted 4 KIPs (KIP-550 to KIP-553) and these are the ones that caught my eye:
- KIP-551: Expose disk read and write metrics: Kafka relies on the file system to store and retrieve records. For that reason, it’s important to monitor disk reads and writes to detect potential performance issues. While it’s possible to retrieve disk metrics via external tools, this KIP aims at providing them as Kafka metrics.
- KIP-553: Enable TLSv1.3 by default and disable all protocols except [TLSV1.2, TLSV1.3].: TLSv1.3 was defined in RFC 8446 in August 2018 and support was added in Kafka via KAFKA-7251. This KIP is the final step to enable its support in Kafka. In addition, this KIP proposes removing support for TLS 1.0 and 1.1 which are now considered obsolete.
- Exploring KEDA: A brief look at KEDA using Kafka triggers
- Should you run Kafka Connect in Distributed or Standalone mode?
- Towards a MongoDB-backed Apache Kafka Streams State Store
Milestones that the project and community achieved in 2019
Releases in 2019
KIPs in 2019
In the past 12 months, the community raised 142 KIPs.
Code and contributors
Over 200 unique contributors made more than 1200 commits in 2019.
git diff --shortstat $(git hash-object -t tree /dev/null)
Committers and PMC
In 2019, five contributors were invited to become Committers:
- Vahid Hashemian
- Bill Bejeck
- Randall Hauch
- Mickael Maison
- John Roesler
Likewise, two Committers also joined the Apache Kafka PMC:
- Matthias J. Sax
- Sriharsh Chintalapan
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.