Digital Developer Conference: Hybrid Cloud 2021. On Sep 21, gain free hybrid cloud skills from experts and partners. Register now

IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

See what's going on in the Kafka community this month


This is the 41st edition of the Kafka Monthly Digest. In this edition, I’ll cover what happened in the Apache Kafka community in June 2021.

For last month’s digest, see Kafka Monthly Digest: May 2021.

Kafka Summit

The Kafka Summit APAC is on July 27 and 28. It is a virtual event with dozens of sessions covering all aspects of the Kafka ecosystem. You can register for free on the Kafka Summit website.

Releases

3.0.0

The release process for Kafka 3.0.0 continued. KIP freeze happened on June 9. The next milestone is feature freeze, currently targeted for the first week of July. To see the list of KIPs included in this release, see the release plan in the wiki.

KIPs

Last month, the community submitted 14 KIPs (KIP-745 to KIP-759, 747 was skipped), and these are the ones that caught my eye.

  • KIP-745: Connect API to restart connector and tasks. When running Connectors in Kafka Connect, the connector itself or any of its tasks can hit an error and enter the FAILED state. At the moment, the Connect REST API allows restarting only the connector or only a single task at a time. This KIP proposes extending the REST API to provide more flexibility when restarting connectors/tasks. It will enable restarting all FAILED tasks or the connector and its tasks in 1 call.

  • KIP-753: ACL authentication, Host field support IP network segment. Authorizations in Kafka are managed via ACLs which follow this format: Principal P is [Allowed/Denied] Operation O From Host H on any Resource R. The Host H portion allows defining an IP for the Principal. The goal of this KIP is to add support subnets instead of using a single IP to simplify the management of ACLs.

  • KIP-758: Remove list of latest producer append information for broker. To handle idempotent and transactional producers, brokers maintain a cache with some data about the 5 latest appends. This cache is complex to maintain and can use a significant amount of memory on brokers. In addition, it only serves as a best effort attempt to return an offset after handling an append. This KIP proposes only keeping data about the last append and using some simple heuristics to handle appends with an invalid sequence number.

Community Releases

In this section, I will cover releases of some community projects. This only includes projects that are open source.

  • Debezium 1.6.0. Debezium is a Change Data Capture platform. This release introduces incremental snapshotting (currently incubating). The Oracle connector can now react to DDL schema changes. Other connectors such as Vitess and Cassandra also include a number of fixes and improvements.

  • akhq 0.18.0. AKHQ is a GUI for Apache Kafka. This new version improves the topic data browser, which allows connecting to a schema registry over TLS and is able to connect to Kafka clusters using OAuth.

  • strimzi-kafka-operator 0.24: Strimzi is a Kubernetes Operator for running Kafka. It now supports Kafka 2.7.1 and the new kafka-kubernetes-config-provider to load configurations from secrets and configmaps. The configuration of Quotas is also improved including support for the recent controller_mutation_rate quota.

Blogs

Get started with Kafka

IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.