Event sourcing: an introduction

Event sourcing is a pattern that records every state change as an immutable event, giving you a complete, replayable history of your system. This article walks through the core concepts, key benefits, and practical considerations—including how Apache Kafka serves as an event store.

Businesses in most industries must react in real time to customer actions and environmental events. Designing applications that are event-driven helps mirror real-world processes. The Event-driven architecture model shows how to build and extend event-driven solutions on cloud-native architectures.

This article assumes you have some familiarity with cloud terminology and Apache Kafka.

What is event sourcing?

Event sourcing is an approach for maintaining the state of business entities by recording each change of state as an event.

Whether a customer buys an item in a shop, or updates a profile online, both actions can be seen as an event. In its most basic form, an event captures a fact. It means that something has happened.

In event sourcing, every state change is captured and stored as an event rather than overwriting the entity's current state. The sequence of state changes is saved instead. The following diagram shows events for three different customers.

The event store must provide a few features to make this approach viable:

• Ordering. New events must be added at the end of the sequence.
• Immutability. Previous events cannot be updated. If an operation must be undone, a new event should be created to revert the previous event.
• Message broker. Applications must be able to subscribe to events.

The goal is to use the event store as the source of truth and allow reprocessing the events to fully rebuild all other states.

For example, a microservice that tracks items bought can replay the event sequence to compute customer IDs and item counts:

Why use event sourcing?

Event sourcing brings some interesting benefits. The first advantage is atomicity, because writing the event is a single operation. Applications can easily ensure the event is correctly recorded. Then each consuming service can handle updating its own state in response to the event. This approach avoids the need to deal with distributed transactions between a database and a message broker.

Using an event store also decouples the different microservices. When emitting an event, an application does not need to know about other applications interested in the event. This approach greatly improves performance and allows the pattern to scale really well.

Finally, event sourcing gives you the ability to use the sequence of events that led to the current state. Whether it is for audit, replayability, or to perform machine learning on past data, the full sequence of events provides more insight than only retaining the current state.

Apache Kafka as the event store

Apache Kafka's rise in popularity is due in large part to its capabilities in event-driven environments. It is extremely well suited for event sourcing because it provides all the required properties, including the ability to scale:

• Kafka Streams is a stream processing library for writing complex processing logic quickly. It supports joining, aggregating, counting, and filtering, so you can focus on business logic.
• Messaging system is a performant, easy-to-use API for emitting and consuming events. Consumer groups let you scale processing microservices as needed.
• Log compaction lets you build snapshots of event streams, reducing startup time. When a topic is compacted, Kafka retains only the last value for each key, so microservices can bootstrap from a snapshot rather than reprocessing all events.

Considerations for event sourcing

You need to take into account a few factors when implementing a solution based on event sourcing.

Because all things evolve, it is very likely the schemas of events change, or new events are added over time. If built without care, you quickly find a situation where every single event update requires updating all consuming applications.

In addition, the initial cost can be high. Each microservice needs some scaffolding to process the stream of events and to update its store. In addition, it’s relatively common to have logic that joins multiple types of events to enrich data. Without a framework or a library, this situation can lead to significant code duplication and some inconsistencies that are hard to debug.

Finally, in practice, you don’t want microservices to reprocess all the events since the beginning of time when you start up an application. It’s important to build regular snapshots that can be used to restart very quickly. As previously mentioned, Apache Kafka helps building snapshots. Otherwise, a database can be used.

Next steps

You now understand event sourcing, which is one of the most common patterns for building event-driven solutions—along with its benefits and implementation considerations.

To try out event sourcing, explore IBM Event Streams, a managed Kafka-as-a-service platform on IBM Cloud.