A review of releases and KIPs for the Apache Kafka project for June 2018!

In this post, I will cover what happened in the Kafka community in June 2018.

Kafka Releases:

June was certainly exciting in term of releases as the community is working on 5 releases! There is of course the next major release, 2.0.0 as mentioned in last month’s post, which should be available in early July.

There are also 4 bugfix releases, 0.10.2.2, 0.11.0.3, 1.0.2 and 1.1.1,  in progress. Votes for all of them have started so if you have time, please test them and vote in their respective thread on the mailing list.

KIPs:

Once again, the community has been very active. 19 KIPs (KIP-309 to KIP-330 but 323 and 329 have been skipped) have been submitted since last month. These are the ones that caught my eye:

KIP-318: Make Kafka Connect Source idempotent
The KIP proposes enabling by default the idempotent producer in the Kafka Source connector for Kafka Connect. While this is already doable by explicitly changing the configuration, the idea is to allow everyone to automatically benefit from the improved performance and delivery semantics.

KIP-320: Allow fetchers to detect and handle log truncation
This KIP’s purpose is to fix a couple of issues when handling Fetch requests. The proposal is to include the epoch from the last Fetch response in each new Fetch request for both consumers (clients) and followers (brokers). Using this new field, brokers will be able to handle correctly an edge case in the replication protocol and also to improve handling of log truncation when unclean leader election is enabled.

KIP-324: Add method to get metrics() in AdminClient
Unlike all the other clients, the AdminClient is not exposing metrics via its API. This KIP aims at finally fixing this by adding a metrics() method allowing users to monitor how the client performing in code.

KIP-325: Extend Consumer Group Command to Show Beginning Offsets and Partition Size
This KIP’s goal is to display the smallest offset in the output of the consumer-groups tool. In addition to showing the partition size (as the largest offset is already displayed) this is also an interesting metric when monitoring Consumer Groups.

KIP-327: Add describe all topics API to AdminClient
Describing all topics is a pretty common operation. Currently, using the AdminClient, users first have to get the list of topics and then pass this list to describeTopics(). This KIP’s objective is to add another describeTopics() method to get all topics descriptions in a single call.

Blogs:


IBM Message Hub is Apache Kafka as a service for IBM Cloud. You can get started at https://console.bluemix.net/docs/services/MessageHub/index.html#messagehub.

 

2 comments on"Kafka Monthly Digest #5: June 2018"

  1. Thanks for the monthly digest! 🙂
    Regarding the Kafka Source connector for Kafka Connect KIP, I think it would need more than just a configuration change given that you would need to explicitly begin and commit/abort a transaction in the source connector. So I’m not sure if it’s already doable by injecting the configuration at the connector level. Is this something supported in Kafka Connect 1.1.1?

    • This KIP is to enable the idempotent producer by default, not the transactional producer. AT the moment, it is still being discussed, so not available yet.

Join The Discussion

Your email address will not be published. Required fields are marked *