In this post, I will cover what happened in the Kafka community in April 2018.
I hope many of you had the opportunity to attend the Kafka Summit in London on the 23rd and 24th of April. I had a great time there and all the sessions I attended were very interesting. The recordings of the keynotes and all the sessions are available online. Jay Kreps also announced the Summit will return to London next year.
Finally, the Call for Papers for the next Summit in San Francisco in October has already started and will run until the 16th of June.
No new releases this month, but preparations for the next one have already started.
The community has decided to call the next release Kafka 2.0. Rajini Sivaram offered to run it and started the process. You can find the release plan on the wiki with the current expected release date for end of June.
Since last month, the community submitted 12 KIPs (KIP-278 to KIP-289). These are the ones that caught my eye:
KIP-279: Fix log divergence between leader and follower after fast leader fail over
This KIP’s goal is to solve an edge case in the replication failover logic which can cause divergence between logs. In addition, it will also fix unclean leader election issues that were identified in KIP-101.
KIP-280: Enhanced log compaction
This KIP aims at making log compaction more flexible. Currently compaction relies on the order that messages were inserted and this can cause issues for workloads that don’t guarantee the order. So the proposal is to allow compaction to append based on the record’s timestamp or headers.
KIP-283: Efficient Memory Usage for Down-Conversion
When handling old clients, newer brokers might have to down convert messages to a format supported by the clients. This process can be expensive in memory and cause OutOfMemory exceptions. This KIP aims at reducing memory used for down conversion and avoiding running out of memory.
KIP-285: Connect Rest Extension Plugin
The REST API endpoint is lacking several features required in enterprise deployments like authentication and authorization. This KIP proposes adding an extension point allowing users to provide plugins that could implement a wide range of features.
IBM Message Hub is Apache Kafka as a service for IBM Cloud. You can get started at https://console.bluemix.net/docs/services/MessageHub/index.html#messagehub.