IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

From releases, to Kafka Improvement Proposals (KIPs), to blogs, check out what's going on in the Apache Kafka community for March 2020.


In this 26th edition of the Kafka Monthly Digest, I’ll cover what happened in the Apache Kafka community in March 2020.

For last month’s digest, see Kafka Monthly Digest: February 2020.

Releases

  • 2.4.1: On March 12, Bill Bejeck released Kafka 2.4.1. This bug fix release contains 39 fixes including 5 blocker JIRAs. For the full list of fixes, you can check the release notes or the release plan on the wiki.

  • 2.5.0: The release process for 2.5.0 is still ongoing. A few issues (KAFKA-9712, KAFKA-9701, KAFKA-9771) were found and required to cut RC1 and RC2. All these JIRAs have been fixed and a new vote for RC3 should start in the next few days. As always the Release Plan is available on the wiki.

KIPs

Last month, the community submitted 11 KIPs (KIP-575 to KIP-585) and these are the ones that caught my eye:

  • KIP-578: Add configuration to limit number of partitions: While adding partitions can be a way to increase application performance, having too many partitions in a cluster can lead to a severe performance degradation. This KIP aims at providing cluster administrators 2 new configurations to define the maximum number of partitions that a broker and the cluster can host. These configurations will support dynamic updates so clusters can still be scaled.

  • KIP-584: Versioning scheme for features: New features are constantly being added to Kafka brokers. Clients currently use the ApiVersions requests, which was introduced in KIP-35, to identify the features that each broker supports. However, this mechanism has a few limitations as it works at the broker level and also does not work well with features that require a specific inter.broker.protocol.version. The goal of this KIP is to provide mechanisms to allow clients to discover features at the cluster level and also to allow administrators to enable features cluster-wide when a new version is deployed. With these mechanisms in place, it should be possible to remove the inter.broker.protocol.version setting and not require “double-roll” upgrades anymore.

  • KIP-585: Conditional SMT: Simple Message Transformations (SMT) is a feature of Kafka Connect. It allows users to declaratively define one-message-at-a-time transformations to perform data cleansing and routing. This KIP proposes adding support for basic logic operators such as topic-matches or has-header to conditionally select which transformations to apply. The goal is to allow building more complex data pipelines with Connect.

Community Releases

In this section, I will cover releases of some community projects. This only includes projects that are Open Source.

  • strimzi-kafka-operator 0.17.0: Strimzi is a Kubernetes Operator for running Kafka. This new version introduces support for Mirror Maker 2 as well as improved configuration options such as defining Java System Properties or public keys for listeners directly in the Custom Resource YAML.

  • Debezium 1.1.0: Debezium is a popular Change Data Capture platform for Kafka. The 1.1 release brings in a number of new features such as Cloud Events support, improved handling for transaction boundaries, and a brand new connector for IBM Db2.

Blogs

Get started with Kafka

IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.