Kafka Monthly Digest – August 2020
See what's going on in the Kafka community this month
In this 31st edition of the Kafka Monthly Digest, I’ll cover what happened in the Apache Kafka community in August 2020.
For last month’s digest, see Kafka Monthly Digest: July 2020.
2.6.0: After 3 release candidates, this new minor version was released on August 6 and brings a number of interesting features. A post was published on the Apache blog, and as always you can find the full list of changes in the release notes or in the release plan on the wiki.
- Switch to TLSv1.3 when using Java 11 or above. TLSv1.2 is still the default with older versions of Java. (KIP-573)
- New APIs to manage quotas. Managing quotas no longer requires access to ZooKeeper and can be done via the Admin API or the
- New metrics tracking disk reads and writes. Administrator can now easily gauge the performance or behavior of brokers. (KIP-551)
- Add context in MetricsReporters. Some context can now be passed to reporters to tag metrics accordingly. (KIP-606)
- Allow listing consumer groups by state. This allows you to see the state of groups via a single command rather than describing them all. (KIP-518)
- Configuration of new topics for Source Connectors. Users can now define the configuration of topics created by Source Connectors. (KIP-158)
- Custom replacement for MaskField SMT. This SMT can inject configurable replacements when masking fields. (KIP-437)
- Configurable HTTP response headers. A new runtime configuration
response.http.headers.configenables defining HTTP headers on responses from Connect’s REST API. (KIP-577)
- Conditional SMTs. Using the new predicate statement, administrators can define conditions for SMTs to be applied. (KIP-585)
- Reduce downtime during rebalance. Streams tasks are only handed over to the new owner once it is ready and this improves several scenarios such as scaling out. (KIP-441
- New metrics for Streams threads and tasks. (KIP-444)
- Improved scalability with “exactly once” semantics. This makes it easier to build EoS applications using a lot of partitions. (KIP-447)
- Remove members with StreamsResetter tool. This tool can now forcibly remove members from a Streams application consumer group. (KIP-571)
2.5.1: John Roesler released 2.5.1 on August 11. This bugfix version contains 71 fixes and improvements including 6 blocker JIRAs. You can check the release notes and the release plan for the full details.
Last month, the community submitted 16 KIPs (KIP-651 to KIP-667, 652 was skipped), and these are the ones that caught my eye.
KIP-656: MirrorMaker2 Exactly-once Semantics. As a set of Source Connectors, MirrorMaker2 offers “at least once” or “at most once” delivery semantics. This KIP aims at supporting “exactly once” semantics by providing a MirrorSink connector and using transactions.
KIP-660: Pluggable ReplicaAssignor. When topics or partitions are created, the Kafka controller has to determine which brokers will host them. At the moment, it is using a round-robin algorithm that can also take into account racks. There are a few cases (scaling or load management, for example) where this algorithm is not optimal. This KIP proposes making this mechanism pluggable to allow administrators to customize the assignment logic for their specific needs.
KIP-664: Provide tooling to detect and abort hanging transactions. When hitting a bug, it’s possible for a transaction to be left hanging. At the moment, this might leave users stuck with no easy way to purge the hung transaction. This KIP’s goal is to improve the management of transactions and provide tools to forcibly abort them if necessary.
KIP-665: Kafka Connect Hash SMT. Simple Message Transformations (SMT) is a feature of Kafka Connect that allows defining one-message-at-a-time transformations. By default, Kafka has a few built-in transformations. This KIP suggests adding a new transformation to hash record keys or values with a specified algorithm.
In this section, I will cover releases of some community projects. This only includes projects that are Open Source.
- Sarama 1.27.0. Sarama is a pure Golang Kafka client. This new release introduces producer and consumer message interceptors that work the same way as in the Java client. It also adds support for manually committing offsets in the consumer.
- Benchmarking Apache Kafka, Apache Pulsar, and RabbitMQ: Which is the Fastest?
- Using Open Policy Agent with Strimzi and Apache Kafka
Get started with Kafka
IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.