Kafka Monthly Digest – May 2020
From releases, to Kafka Improvement Proposals (KIPs), to blogs, check out what's going on in the Apache Kafka community for May 2020.
In this 28th edition of the Kafka Monthly Digest, I’ll cover what happened in the Apache Kafka community in May 2020.
For last month’s digest, see Kafka Monthly Digest: April 2020.
2.6.0: On May 5, Randall Hauch volunteered to run the 2.6.0 release. KIP freeze happened on May 20, and feature freeze was on May 27. The release date is currently expected for the end of June. For the full details, see the release plan in the wiki.
2.5.1: On May 29, John Roesler volunteered to be the release manager for 2.5.1. The release plan for this bugfix version is available on the wiki.
Last month, the community submitted 16 KIPs (KIP-604 to KIP-619), and these are the ones that caught my eye.
KIP-604: Remove ZooKeeper Flags from the Administrative Tools. As work to remove Zookeeper is under way, this KIP is for removing the
--zookeeperflag from all command line tools in the next major release, 3.0. This flag is already deprecated, and all tools can use the
--bootstrap-serverflag to directly connect to Kafka.
KIP-612: Ability to Limit Connection Creation Rate on Brokers. Receiving new client connections uses some CPU on brokers. While this is normally negligible, a connection storm can apply significant load onto brokers. Connection storms can be caused by “well-behaved” clients, for example, when a large application with dozens of clients restarts. This KIP aims at providing a new configuration,
max.connection.creation.rate, to enable cluster administrators to limit client connections creations.
KIP-613: Add end-to-end latency metrics to Streams. At the moment, Kafka Streams does not provide end-to-end latency metrics. This makes it hard to build and operate realtime applications. This KIP aims at adding a number of end-to-end latency metrics to understand how long it takes for records to flow through a topology.
KIP-618: Atomic commit of source connector records and offsets. This KIP’s goal is to strengthen the delivery guarantees for source connectors. In addition to source records, source connectors also write their current offsets to Kafka. Currently, these 2 writes are done independently. This can cause some record duplication, for example, if a source record is produced but the associated offset is not. This KIP proposes grouping these 2 writes in a transaction to provide
exactly-oncesemantics for source connectors.
In this section, I will cover releases of some community projects. This only includes projects that are Open Source.
Strimzi 0.18.0. Strimzi is a Kafka Operator for Kubernetes. This new release adds support for Kafka 2.4.1 and 2.5.0. It also improves monitoring via new metrics for Clusters, Users, and Topics, and provides a new Grafana dashboard for Operators.
Azkarra Streams 0.7.0. Azkarra is a framework to help build streaming microservices using Kafka Streams. In addition to bugfixes, this new release introduces a few new features, such as improved consumption monitoring via a new REST endpoint and REST extensions that allow registering custom JAX-RS resources. You can find all the details in Introducing Azkarra Streams 0.7.
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.