Kafka Monthly Digest – June 2021
See what's going on in the Kafka community this month
This is the 41st edition of the Kafka Monthly Digest. In this edition, I’ll cover what happened in the Apache Kafka community in June 2021.
For last month’s digest, see Kafka Monthly Digest: May 2021.
The Kafka Summit APAC is on July 27 and 28. It is a virtual event with dozens of sessions covering all aspects of the Kafka ecosystem. You can register for free on the Kafka Summit website.
The release process for Kafka 3.0.0 continued. KIP freeze happened on June 9. The next milestone is feature freeze, currently targeted for the first week of July. To see the list of KIPs included in this release, see the release plan in the wiki.
Last month, the community submitted 14 KIPs (KIP-745 to KIP-759, 747 was skipped), and these are the ones that caught my eye.
KIP-745: Connect API to restart connector and tasks. When running Connectors in Kafka Connect, the connector itself or any of its tasks can hit an error and enter the
FAILEDstate. At the moment, the Connect REST API allows restarting only the connector or only a single task at a time. This KIP proposes extending the REST API to provide more flexibility when restarting connectors/tasks. It will enable restarting all
FAILEDtasks or the connector and its tasks in 1 call.
KIP-753: ACL authentication, Host field support IP network segment. Authorizations in Kafka are managed via ACLs which follow this format:
Principal P is [Allowed/Denied] Operation O From Host H on any Resource R. The
Host Hportion allows defining an IP for the Principal. The goal of this KIP is to add support subnets instead of using a single IP to simplify the management of ACLs.
KIP-758: Remove list of latest producer append information for broker. To handle idempotent and transactional producers, brokers maintain a cache with some data about the 5 latest appends. This cache is complex to maintain and can use a significant amount of memory on brokers. In addition, it only serves as a best effort attempt to return an offset after handling an append. This KIP proposes only keeping data about the last append and using some simple heuristics to handle appends with an invalid sequence number.
In this section, I will cover releases of some community projects. This only includes projects that are open source.
Debezium 1.6.0. Debezium is a Change Data Capture platform. This release introduces incremental snapshotting (currently incubating). The Oracle connector can now react to DDL schema changes. Other connectors such as Vitess and Cassandra also include a number of fixes and improvements.
akhq 0.18.0. AKHQ is a GUI for Apache Kafka. This new version improves the topic data browser, which allows connecting to a schema registry over TLS and is able to connect to Kafka clusters using OAuth.
strimzi-kafka-operator 0.24: Strimzi is a Kubernetes Operator for running Kafka. It now supports Kafka 2.7.1 and the new kafka-kubernetes-config-provider to load configurations from secrets and
configmaps. The configuration of
Quotasis also improved including support for the recent
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.