The amount of data being produced every day is growing all the time, and a large amount of this data is in the form of events. Whether it be updates from sensors, clicks on a website, or even tweets, applications are bombarded with a never-ending stream of new events. So, how can we architect our applications to be more reactive and resilient to the fluctuating loads and better manage our thirst for data? Many companies are adopting Apache Kafka as a key technology to achieve this.
In this article, learn all about the Kafka configurations you will need to consider to ensure your application is as responsive, elastic, resilient and reactive as possible.
Why use Apache Kafka?
Apache Kafka is an open-source, distributed streaming platform that is perfect for handling streams of events. You can learn more about what Kafka is from this technical article, “What is Apache Kafka?.”
Although Kafka is a fantastic tool to use when dealing with streams of events, if you need to serve up this information in a reactive and highly responsive manner, Kafka needs to be used in the right way with the best possible configuration.
Why use a reactive architecture?
The term “Reactive systems” refers to an architectural style that enables applications composed of multiple microservices working together as a single unit. The aim of this architecture style is to enable applications to better react to their surroundings and one another, which manifests in greater elasticity when dealing with ever-changing workload demands and resiliency when components fail.
The Reactive Manifesto helps to define the key characteristics that are involved in creating a truly reactive system: responsive, resilient, elastic, and message-driven.
You can learn more about what reactive systems are and how to design and build one using the learning paths and content patterns in this “Getting started with Reactive Systems” article. Or, for a more in depth explanation, you can read the report, “Reactive Systems Explained.”
How Kafka fits into reactive systems
Reactive systems rely on a backbone of non-blocking, asynchronous message-passing, which helps to establish a boundary between components that ensures loose coupling, isolation, and location transparency. Non-blocking communication allows recipients to only consume resources while active, which leads to less system overhead. Employing explicit message-passing enables load management, elasticity, and flow control by shaping and monitoring the message queues in the system and applying back-pressure when necessary.
Kafka is a great tool to enable the asynchronous message-passing that makes up the backbone of a reactive system. So, how do we configure Kafka to also enable resiliency and elasticity within our applications so that it can effectively respond to the events it consumes?
How to configure Kafka for reactive systems
Kafka is highly configurable, so it can be tailored depending on the application. When building reactive systems, we need to consider resiliency and elasticity and which configuration values we need to be aware of to enable these. By enabling our application to be message-driven (as we already know Kafka enables), and resilient and elastic, we can create applications that are responsive to events and therefore reactive.
In regards to resiliency, Kafka already has natural resiliency built in, using a combination of multiple, distributed brokers that replicate records between them. However, using a set of distributed brokers alone does not guarantee resiliency of records from end-to-end. To achieve this resiliency, configuration values such as acknowledgements, retry policies, and offset commit strategies need to be set appropriately in your Kafka deployment.
Acknowledgements and retries
Kafka can be configured in one of two ways for record delivery: “at least once” and “at most once.” If your applications are able to handle missing records, “at most once” is good enough. However, when dealing with business critical messages, “at least once” delivery is required. These record delivery options are achieved by setting the
retries configuration options of producers.
acks (acknowledgement) configuration option can be set to
0 for no acknowledgement,
1 to wait for a single broker, or
all to wait for all of the brokers to acknowledge the new record. The Producer API also allows configuration of the number of retries to attempt if the producer times out waiting for the acknowledgement from the brokers.
For “at most once” delivery of records, both
retries can be set to
0. This is a “fire-and-forget” approach. For “at least once” delivery (the most common approach used in reactive applications)
acks should be set to
all. This configuration does however introduce higher latency, so depending on your application you may settle for
acks set to
1 to get some resiliency with lower latency.
To get “at least once” delivery, setting
all is not enough. Applications also need to deal with the records that have failed to reach the brokers. This resiliency can be achieved by increasing the number of
retries or using custom logic. Note that allowing retries can impact the ordering of your records.
In regards to the consumers, it’s the strategy of committing offsets that matters the most. The committed offset denotes the last record that a consumer has read or processed on a topic. These offsets are committed to Kafka to allow applications to pick up where they left off if they go down.
If auto-commit is enabled (which then commits the latest offset on a timer basis), you might lose records because the length of time between commits might not be sufficient. When this is the case, an application can go down after the offset has been committed but before the record was fully processed. When the application is restarted, it starts consuming records after the lost record due to the offset already being committed for that particular record. As a result, the unprocessed record is skipped and has been effectively lost.
If auto-commit is disabled, you will be able to control exactly when the consumer commits the latest offset. In a reactive system, manual commit should be used, with offsets only being committed once the record is fully processed.
There is in-built scalability within Kafka. For example, brokers and partitions can be scaled out. Unfortunately, those brokers and partitions cannot be scaled back down, at least not safely in an automated fashion. It could be argued that Kafka is not truly elastic, but using Kafka does not prevent you from creating a system that is elastic enough to deal with fluctuating load. Since Kafka is designed to be able to handle large amounts of load without using too much resource, you should be focusing your efforts on building elastic producers and consumers.
Once a producer application has been written, you do not need to do anything special to be able to scale it up and down. The main consideration is how to scale your producers so that they don’t produce duplicate messages when scaled up. This will be handled down in the business logic within your application, so it is your responsibility to consider this when writing the application.
When scaling consumers, you should make use of consumer groups. Consumers can collaborate by connecting to Kafka using the same group ID, where each member of the group gets a subset of the records on a particular topic. This subset will be in the form of one or more partitions. Since consumers in a group do not want an overlap of the records they process, each partition is only accessible to one consumer within a consumer group. If you want to scale up to have more consumers than the current number of partitions, you need to add more partitions. However, increasing the partition count for a topic after records have been sent removes the ordering guarantees that the record keys provide. Therefore, if you care about ordering, you should think carefully about the number of partitions you initially instantiate for each topic.
Kafka reactive frameworks
Apache Kafka provides a Java Producer and Consumer API as standard, however these are not optimized for Reactive Systems. To better write applications that interact with Kafka in a reactive manner, there are several open-source Reactive frameworks and toolkits that include Kafka clients:
Vert.x is a polyglot toolkit, based on the reactor pattern, that runs on the JVM. It is non-blocking and event-driven and includes a distributed event bus within it that helps to keep code single-threaded. The Vert.x Kafka Client within this toolkit enables connection to Apache Kafka.
MicroProfile Reactive Messaging is a specification that is part of the wider cross-vendor MicroProfile framework. With MicroProfile Reactive Messaging, you annotate application beans’ methods and, under the covers, OpenLiberty can then convert these to reactive streams-compatible publishers, subscribers and processors and connects them up to each other. The Kafka Connector, within the provided Connector API library, enables connection to external messaging systems including Apache Kafka. For more information on this and how to effectively use MicroProfile Reactive Messaging check out this useful blog.
Project Reactor is a reactive library also based on the Reactive Streams Specification that operates on the JVM. Reactor Kafka is an API within project reactor that enables connection to Apache Kafka.
Alpakka is a library built on top of the Akka Streams framework to implement stream-aware and reactive integration pipelines for Java and Scala . Alpakka Kafka Connector enables connection between Apache Kafka and Akka Streams.
We have built an an open source sample starter Vert.x Kafka application which you can check out in the ibm-messaging / kafka-java-vertx-starter GitHub repository. Read more about our journey, transforming our kafka starter app into a Vert.x reactive app in this tutorial, “Experiences writing a reactive Kafka application.
Summary and next steps
Kafka has become the de-facto asynchronous messaging technology for reactive systems. However, using Kafka alone is not enough to make your system wholly reactive. When writing applications, you must consider how your applications integrate with Kafka through your producers and consumers. By taking the time to configure your applications appropriately, you can make the most of the built-in resiliency and scalability that Kafka offers.
If you are looking for a fully supported Apache Kafka offering, check out IBM Event Streams, the Kafka offering from IBM. Try Event Streams on IBM Cloud as a managed service, or deploy your own instance of Event Streams in IBM Cloud Pak for Integration on Red Hat OpenShift Container Platform. Event Streams in IBM Cloud Pak for Integration adds on valuable capabilities to Apache Kafka including powerful ops tooling, a schema registry, award-winning user experience, and an extensive connector catalog to enable a connection to a wide range of core enterprise systems.