Kafka Monthly Digest – August 2021
See what's going on in the Kafka community this month
This is the 43rd edition of the Kafka Monthly Digest. In this edition, I’ll cover what happened in the Apache Kafka community in August 2021.
For last month’s digest, see Kafka Monthly Digest: July 2021.
The Kafka Summit Americas is on September 14 and 15. It is a virtual event with dozens of sessions covering all aspects of the Kafka ecosystem. You can register for free on the Kafka Summit website.
Kafka 3.0.0 is currently in the stabilization period following code freeze. After a few more Blocker JIRAs (KAFKA-13198, KAFKA-13219), Konstantine published the first Release Candidate, 3.0.0 RC0, on August 27. A vote on this RC is currently on going.
On August 12, David Jacot volunteered to run this bugfix release. The release date is currently targetted for September. The release plan is available on the wiki.
Last month, the community submitted 5 KIPs (KIP-766 to KIP-770), and these are the ones that caught my eye.
KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC. Kafka supports OAuth authentication since 2.0 (KIP-255). However, KIP-255 only introduced a framework and organizations have to provide their own implementations. This KIP proposes adding a production ready implementation of the interfaces defined in KIP-255 to authenticate with Open ID identity providers.
KIP-770: Replace “buffered.records.per.partition” with “input.buffer.max.bytes”. This KIP aims at controlling the memory used by a Kafka Streams application. It proposes adding a new configuration
input.buffer.max.bytesto specify the total memory that can be used to buffer records. The existing configuration
buffered.records.per.partitionwhich only controlled memory per partition will be deprecated.
In this section, I will cover releases of some community projects. This only includes projects that are open source.
kafka-connect-file-pulse 2.2.0. FilePulse is a connectors to stream any file into Kafka. This new version improves scenarios with a very large number of small files, improves support for XML files and introduces a new pluggable Partitioner interface to distribute files across tasks.
kcat 1.7.0. Previously called kafkacat, kcat is a CLI client for Kafka. This release improves multibytes delimiters and adds support for incremental rebalancing.
kcctl 1.0.0-EA. kcctl is a command line client for Kafka Connect. It allows to easily interact (deploy, stop, restart, describe, etc) with connectors.
- Real-time Einstein Insights Using Kafka Streams
- Leveraging Graph Database for detection of Fair Play Violation at Dream11
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.