In this post, I will cover what happened in the Kafka community in March 2018.
We have 2 new releases: 1.1.0 and 1.0.1!
– Kafka 1.1.0:
It was released on the 29th of March. It is slightly late according to the time-based release plan (it was planned for February) due to a late issue with Java 9 and 10 which required to cut an RC4! In addition to the regular clean-ups and fixes, it contains a lot of interesting new features:
– Dynamic Broker configuration: A number of broker configurations can now be updated at runtime.
– Better JBOD support: Replicas can now be moved between log directories. That allows to easily add/remove disks to brokers or balance throughput across disks.
– Replication improvements: Partitions with no or very little traffic now use a lot less resources as 1.1.0 brokers use incremental fetch requests that are lighter than normal fetch requests.
– Controller improvements: The controller rework carries on in this release as it now uses an asynchronous Zookeeper client that enables to pipeline requests. This is especially important as the controller can do a ton of small writes (1 per partition in some cases), so having a pipelined asynchronous client avoids waiting an entire round trip before doing another write.
– Delegation token-based authentication: Kafka can now use delegation tokens to perform authentication. This enables building deployments where only a few nodes need to directly authenticate and can distribute tokens.
– Connect improvements: Connect now supports message headers and topic regex for sink connectors. Also, the REST endpoint can be configured to use TLS and access via HTTPS.
– Streams improvements: Streams also received a number of improvements including customizable error handling for produce failures. It can also handle broker unavailability better and attempt retries like the other clients.
– Kafka 1.0.1:
As hinted in last post, 1.0.1 was released on the 6th of March. Since 1.0.0, it includes 101 commits fixing 52 JIRAs.
Because it is a bug fix release, it contains no new features but simply a number of fixes and stability improvements. The full release notes are available on the Apache website.
Once again, the community has been very active. 13 KIPs (KIP-265 to KIP-277) have been submitted since last month. These are the ones that caught my eye:
KIP-266: Add TimeoutException to KafkaConsumer.position()
Some of the Consumer methods can block forever in some scenarios. This KIP aims at adding a timeout argument to these methods to avoid hanging consumers.
KIP-270: A Scala wrapper library for Kafka Streams
Using the Streams API from Scala can be cumbersome. The goal of this KIP is to provide a wrapper as part of the Streams library to improve its usability from Scala.
KIP-272: Add API version tag to broker’s RequestsPerSec metric
Brokers support clients using older versions. This however comes at a cost on the broker side as they may have to down-convert messages when a client is using an old message format. This KIP proposes to tag requests metrics to include the client version to allow administrators to track what portion of the clients have been upgraded following a broker update.
KIP-273: Kafka to support using ETCD beside Zookeeper
Currently Zookeeper is required in order to have a Kafka cluster. This KIP aims at adding support for a different consensus library, ETCD. A similar KIP was already proposed a couple of year ago (KIP-30) but was rejected by the community.
KIP-277: Fine Grained ACL for CreateTopics API
Disclaimer, I’m one of the co-authors. This KIP aims a making the Access Control Lists (ACLs) required in order to create topics more granular and be in accord with the ACLs required to delete topics.
Blogs and Resources:
IBM Message Hub is Apache Kafka as a service for IBM Cloud. You can get started at https://console.bluemix.net/docs/services/MessageHub/index.html#messagehub.