In this series of posts, I will provide a digest of what happens in the Apache Kafka community on a monthly basis. It will cover releases, Kafka Improvement Proposals (KIP), and interesting blog articles and resources.
In this first post, I will cover both January and February.
No new versions were released but the release process was started for both 1.0.1 and 1.1.0.
– 1.0.1: This is the first bugfix release since 1.0 and it contains over 100 commits and fixes 43 JIRAs. Ewen Cheslack-Postava started the release process on the 16th of January. Two more release candidates later it’s finally 1.0.1 RC2 that became 1.0.1. More in next month’s digest!
– 1.1.0: Damian Guy volunteered to run the 1.1.0 release. The vote for RC0 started on the 24th of February but a couple of JIRAs (KAFKA-6593 and KAFKA-6111) required to cut another RC. Currently the vote on RC2 is still ongoing.
There has been a huge amount of activity on KIPs by the community as 16 KIPs have been submitted! I’m not going to cover all of them, but these are the ones that caught my eye:
KIP-247: Add public test utils for Kafka Streams
This KIP proposes adding a public API to help test Kafka Streams application. Complex topologies can be hard to get right and it’s important to provide tools, so users can validate and properly test their code.
KIP-250: Add Support for Quorum-based Producer Acknowledgment
When producing messages with acks=-1, the 99.9% latency can suffer as it depends on all the in-sync replicas to acknowledge messages. This KIP’s goal is to reduce this latency while keeping strong message delivery guarantees.
KIP-252: Extend ACLs to allow filtering based on IP ranges and subnets
Currently ACLs can be defined on IPs but it requires to specifically list them all. This KIP aims at providing IP range and subnet filters to simplify ACL management.
KIP-253: Support in-order message delivery with partition expansion
Kafka supports adding partitions to topics for a very long time. However, this comes with a catch for keyed messages: key-partition affinity is not kept. This KIP aims at solving this issue and also allowing users to remove partitions.
KIP-255: OAuth Authentication via SASL/OAUTHBEARER
This KIP proposes adding support for the OAUTHBEARER Sasl mechanism. Currently Kafka supports the following SASL mechanisms: Kerberos, Plain and Scram. OAuth is an industry standard that might be supported by Kafka in the near future.
KIP-257: Configurable Quota Management
Quotas are an important feature for cluster administrators to prevent network and CPU saturation. They are however not very flexible. This KIP proposes making the Quotas logic pluggable hence allowing administrators to provide custom logic. That would enable having Quotas for groups of users or making Quotas scale when topics/partitions are created/deleted for example.
Kafka Blogs and Resources
- Building Reliable Reprocessing and Dead Letter Queues with Kafka
- Learning Kafka Streams in a Week (or less)
The full agenda for the next Kafka Summit has been announced. It will be on the 23rd and 24th of April and it’s already sold out. For the full details, visit the Kafka Summit website.