Win $20,000. Help build the future of education. Answer the call. Learn more

IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

See what's going on in the Kafka community this month


This is the 39th edition of the Kafka Monthly Digest! In this edition, I’ll cover what happened in the Apache Kafka community in April 2021.

For last month’s digest, see Kafka Monthly Digest: March 2020.

Releases

2.8.0

John Roesler released 2.8.0 on April 19. A post was published on the Apache blog, and as always you can find the full list of changes in the release notes or in the release plan on the wiki.

Kafka 2.8.0 introduces the option to run Kafka without ZooKeeper in early access (KIP-500). This mode is named KRaft, for Kafka Raft. This is not for production as there is currently no migration path and many features are still disabled, but if you want to try it out in a test cluster before 3.0.0, see the KRaft README.

This new minor version brings a number of interesting features:

  • Brokers/clients

    • Mutual TLS can now be enabled on SASL listeners. (KIP-684)
    • Quotas can now be defined on the connection creation rate. (KIP-612)
  • Connect

    • The computed configuration of Connector tasks can now be retrieved via the REST API. (KIP-661)
  • Streams

    • Applications can register a Streams exception handler that can be used to halt or recover in case of errors. (KIP-671)
    • A new API enables starting or stopping Streams threads to scale or to recover from failures. (KIP-663)

2.6.2

On April 21, Sophie Blee-Goldman released 2.6.2. This bugfix version addresses 35 JIRAs including a handful of blockers. For more details, see the release notes or the release plan.

2.7.1

I published the third release candidate, 2.7.1-RC2, on April 8. The vote passed on May 6 and I’ll release 2.7.1 in the next few days.

3.0.0

Kafka 3.0.0 is the next major release. As mentioned last month, the next phase is KIP freeze. This is currently targeted for early June. See the release plan in the wiki for all the details.

KIPs

Last month, the community submitted 9 KIPs (KIP-729 to KIP-737), and these are the ones that caught my eye.

  • KIP-730: Producer ID generation in KRaft mode. Until now, producer IDs generated for idempotent and transactional producers were stored in ZooKeeper. With KIP-500 and ZooKeeper going away, this KIP proposes a new mechanism to generate and manage producer IDs. A new API, AllocateProducerIds, will be used by brokers to request producer IDs from the controller.

  • KIP-731: Record Rate Limiting for Kafka Connect. This KIP’s goal is to provide a quota mechanism to Kafka Connect. This will allow administrators to define a maximum record rate, consume rate and produce rate per task per connector. This will useful when running multiple connectors in a Kafka Connect cluster to avoid them fighting for resources. This could also be used to prevent Connect from overloading one of the systems it connects to, including Kafka!

  • KIP-734: Improve AdminClient.listOffsets to return timestamp and offset for the record with the largest timestamp. This KIP proposes extending listOffsets() to also optionally return the largest timestamp alongside the offset. The largest timestamp can be good indication of “liveness” for a partition.

Community Releases

In this section, I will cover releases of some community projects. This only includes projects that are open source.

  • Debezium 1.5.0: As usual there are a lot of new features and fixes in this new release of Debezium. Most notably, there’s a brand new implementation of the MySQL connector (with transaction metadata support) and the LogMiner-based Oracle connector is now marked as stable.

Blogs

Kafka Summit

The Kafka Summit Europe is on May 11 and 12. It is a virtual event with over 100 sessions covering all aspects of the Kafka ecosystem. You can register for free on the Kafka Summit website.

Get started with Kafka

IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.