Kafka Monthly Digest – May 2021
See what's going on in the Kafka community this month
This is the 40th edition of the Kafka Monthly Digest. In this edition, I’ll cover what happened in the Apache Kafka community in May 2021.
For last month’s digest, see Kafka Monthly Digest: April 2020.
Kafka 3.0.0 is the next major release. As mentioned last month, the next phase is KIP freeze and it is targeted for early June. See the release plan in the wiki for all the details.
Last month, the community submitted 6 KIPs (KIP-738 to KIP-744, KIP-742 was skipped), and these are the ones that caught my eye.
KIP-738: Removal of Connect’s internal converter properties. Connect’s internal converter properties (
internal.value.converter) have been deprecated since Kafka 2.0.0. This KIP is proposing to remove these settings because users often confuse them with the actual data converters and setting these properties incorrectly can lead to worker failures.
KIP-739: Block Less on KafkaProducer#send.
KafkaProducer.send()) returns a
Futurebut in some cases (when the buffer is full or when a metadata refresh is required) it can also block. This can cause issues in fully asynchronous frameworks so this KIP aims at introducing a
send()method that never blocks.
With 3.0.0 approaching, Kafka Streams is taking the opportunity to do some cleanup with the following KIPs:
KIP-741: Change default serde to be null. At the moment, by default, Streams uses a byte array Serdes. However, in practice, it’s often preferred for users to explicitly specify Serdes in order to immediately catch serialization issues. Hence this KIP proposes removing the default Serdes.
KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics version config. KIP-444 in Kafka 2.4 improved Streams operational experience, which included adding labels to many metrics, and to not break compatibility, a configuration was introduced to keep the old behavior. This KIP proposes removing that configuration to simplify the maintenance of logic handling metrics.
In this section, I will cover releases of some community projects. This list only includes projects that are open source.
- Librdkafka 1.7.0. Librdkafka is a Kafka client in C/C++. This new version improves the reliability of idempotent and transactional producers and adds support for the OpenSSL Engine. The Confluent clients in (Golang, Python and .Net) based on librdkafka have also been updated to use this new release.
- strimzi-kafka-operator 0.23: Strimzi is a Kubernetes Operator for running Kafka. Kafka 2.6.2 and 2.8.0 are now supported. As always, this release includes loads of incremental improvements such as support for configuring connection count and creation rate quotas and configuring a custom Authorizer.
- Sarama 1.29.0. Sarama is a pure Golang Kafka client. This new release adds support for Kafka 2.8.0. It’s also now possible to alter SCRAM credentials via
UpsertUserScramCredentialsand the handling of zstd batches has been improved.
- Exploring ZooKeeper-less Kafka
- Topic naming conventions: How do I name my topics? 5 recommendations with examples
- Getting Started with AsyncAPI: How to Describe you Kafka Cluster
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.