Digital Developer Conference: Hybrid Cloud. On Sep 22 & 24, start your journey to OpenShift certification. Free registration

IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

From releases, to Kafka Improvement Proposals (KIPs), to blogs, check out what's going on in the Apache Kafka community for December 2019.


In this 23rd edition of the Kafka Monthly Digest, I will cover what happened in the Apache Kafka community in December as well as some of the milestones the project and community reached in 2019!

For last month’s digest, see Kafka Monthly Digest: November 2019.

Releases

After four Release Candidates, Manikumar Reddy released Apache Kafka 2.4.0 on December 16. This new minor version brings a number of interesting features:

Broker/Consumer/Producer

  • New Java authorizer Interface. This new interface provides a richer context to authorizers and also supports batching (KIP-504)
  • Administrative API for replica reassignment. Reassignments can now be triggered via the AdminClient and are now incremental and cancelable (KIP-455)
  • Support for incremental cooperative rebalancing to the consumer rebalance protocol. This new incremental protocol provides shorter rebalances for heavy, stateful consumer applications such as Kafka Streams (KIP-429)
  • Allow consumers to fetch from closest replica. Consumers can now fetch from follower replicas to avoid cross datacenter traffic if possible (KIP-392)
  • Sticky Partitioner. This new partitioner is optimized for message batching and can significantly improve performance for some workloads (KIP-480)
  • ZooKeeper 3.5 support
  • Scala 2.13 support

Connect

  • MirrorMaker 2.0. This new tool allows to mirror multiple clusters and create complex replication topologies. It also supports dynamic topic creation/updates and offsets translation (KIP-382)
  • Securing internal Connect REST endpoints. Authentication and Authorizations can be configured to secure the Connect REST API (KIP-507)
  • Extend Connect Converter to support headers (KIP-440)

Streams

  • Support for non-key joining in KTable. A foreign key can now be used for table to table joins (KIP-213)
  • Allow to define custom processor names with KStreams DSL (KIP-307)
  • TopologyTestDriver test input and output usability improvements (KIP-470)

A blog post with more details was published on https://blogs.apache.org/kafka/date/20191216 and as always, the full release notes are available on apache.org. The release plan is on the wiki.

KIPs

Last month, the community submitted 4 KIPs (KIP-550 to KIP-553) and these are the ones that caught my eye:

  • KIP-551: Expose disk read and write metrics: Kafka relies on the file system to store and retrieve records. For that reason, it’s important to monitor disk reads and writes to detect potential performance issues. While it’s possible to retrieve disk metrics via external tools, this KIP aims at providing them as Kafka metrics.

Blogs

Milestones that the project and community achieved in 2019

Releases in 2019

The project followed its time-based release plan. Consequently, it released three major versions: 2.2.0, 2.3.0 and 2.4.0.

In addition, four bugfixes were also released: 2.1.1, 2.2.1, 2.3.1, 2.2.2.

Releases in 2019

KIPs in 2019

In the past 12 months, the community raised 142 KIPs.

KIPs

Code and contributors

Over 200 unique contributors made more than 1200 commits in 2019.

Lines of code

Generated using git diff --shortstat $(git hash-object -t tree /dev/null)

Committers and PMC

In 2019, five contributors were invited to become Committers:

  • Vahid Hashemian
  • Bill Bejeck
  • Randall Hauch
  • Mickael Maison
  • John Roesler

Likewise, two Committers also joined the Apache Kafka PMC:

  • Matthias J. Sax
  • Sriharsh Chintalapan

Get started with Kafka

IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.