Running a compliant Kafka service

We frequently see with our enterprise customers that the cost of providing a compliant Kafka service is incredibly onerous. Different business areas (banking, healthcare, government) all have their own standards and regulations that must be met to protect their own and their customers' assets. These regulations are important, and going forward, there will be more, not less! This onerous problem is not going away!

Imagine the following scenario: A while ago, your boss came to you asking for a solution to his messaging problem. Your answer: "Apache Kafka!" You easily get this up and running, and everyone is happy. More people within your company see how useful Kafka is and how it can solve their problems too. You now find that there are several instances of Kafka being run within different departments across your company. Some have even started using Kafka in production! All is good. However, you then get a visit from your company's security officer asking whether you're storing customer data within Kafka, and if so, how is the data protected? You are storing customer data and you're sure it is protected, so everything is good, right?

Unfortunately not. You need to be able to prove it!

A simple way to answer this is showing who has access to the customer data. You'd already thought of this and have evidence to show that only those people who have a business need can access the production systems and no one has access to the data stored within Kafka. Phew, you're done!

Unfortunately not. Your security officer comes back, and the questions are relentless:

• Do you back up the customer data and who has access to this?
• How do you ensure customers can only access their data but not other customers' data?
• What capacity or network monitoring do you have in place?
• How do you ensure that customer data is deleted when they no longer wish to use to your service?
• How? How? How?

Not only do the questions keep coming, the follow ups are "and can you prove it?" Suddenly you're swamped. This is a full time job and not what you signed up for!

If this scenario feels familiar, or you are looking to expand your business into areas which require such standards, you are not alone.

What actually is compliance?

Everyone wants reassurance that the data they put through a Kafka service is protected and secure. There are many different compliance standards out there, which if you are certified for them demonstrate the required level of protection. Which compliance standards you must adhere to depends on the type of business you or your customers are in.

Some of the more well known compliance standards include:

• System and Organization Controls (SOC1, SOC2, and SOC3)
• General Data Protection Regulation (GDPR) within the European Union
• The Health Insurance Portability and Accountability Act of 1996 (HIPAA)
• International Organization for Standardization (ISO 27001)
• Payment Card Industry Data Security Standard (PCI DSS)

While there is commonality across these standards, they all have slightly different requirements. Some standards require policies about operator access to customer data in production, others require technical evidence that certain criteria are met. Some need recertification on an annual basis, others require evidence from any point within the last six months.

As someone who must manage a Kafka service, you need to be certified for the compliance standards that are relevant to you or your customers.

What is the cost of compliance?

It may be that today you have a small number of people managing the deployment and maintenance of Kafka. You may be deploying Kafka on top of Kubernetes and pushing updates when there are new Kafka versions. It is taking a small number of people to manage and operate the system. However, with compliance in the mix, this number dramatically increases!

Compliance requires significant, continual investment including these tasks:

• Answering a broad range of questions
• Understanding how the standards translate to your business and development processes
• Gathering the evidence for all those "How?" questions
• Managing the interactions with auditors
• Ensuring that the processes are followed at all times

Even after you have achieved the rubber stamp and you are officially certified for a particular compliance standard, your work isn't over. You have to be able to demonstrate that as a business you're continually following the processes you said you were. It never stops!

What is the cost of not doing compliance?

The cost of not doing compliance is huge. There have been well publicized examples where companies have been found to breach compliance standards. Not only does this come with a significant and immediate financial cost, there is the non-tangible cost of bad press and the impact this has on your brand. Data security is highly important and the cost of not doing it can be crippling from both a financial and reputation point of view.

Are you sure compliance requires more investment?

One area of compliance, particularly SOC1 and SOC2, is demonstrating that security issues such as vulnerabilities and new operating system patches are acknowledged, are resolved, and fixes are deployed across all production systems within a very short timeframe. This requires that you implement or use a vulnerability management tool to notice the vulnerabilities in the first place, a tracking tool to enable you to acknowledge the problem and record the investigation and resolution to the problem, and then having a fast pipeline to deploy the fix out to production.

Compliance requires:

• Process investment
• Development investment (to pull in or fix the vulnerabilities)
• Project management investment (to keep on top of the vulnerability status)

Security vulnerabilities are being discovered all the time, sometimes at the rate of several per day. Keeping on top of these vulnerabilities can become a full time job for many people within the team. Moreover, this is not a one-off job! It requires continuous effort to maintain compliance readiness.

How can I do this easily?

If you want to do this yourself, there is no shortcut to handling compliance. The very purpose of compliance is to ensure that processes are in place for the protection and security around managing data, which means compliance processes are very detailed and very thorough.

To be successful, you need to invest in it. Invest the time both in developing automation to capture and alert on the right things and in developing processes that are easy to follow and have the buy in from the development and operational teams. The more you automate, the more you take out the risk of human error, and the easier it is to produce the required evidence. You also need to invest in people to manage compliance, both from a team and company perspective. People need to be in place to liaise with the auditors, manage the data requests, and keep on top of the processes. If this seems a lot, it is.

However, there is a much easier way to handle compliance: Switch from running your own Kafka service to using a managed Kafka service! IBM Event Streams for IBM Cloud is a managed service that handles all the compliance and deployment for you.

It is very easy to get started, see our Getting Started with IBM Event Streams for IBM Cloud Tutorial and our IBM Event Streams for IBM Cloud Plans for the different plans we offer along with the compliance certifications they hold. Most importantly, we ensure 99.95% availability 24x7 and we manage all the headaches of compliance and deployment for you!

Summary

Compliance is important and running a compliant Kafka service requires a significant level of continual investment. Switching to using IBM Event Streams for IBM Cloud will make this all just go away and be our problem, not yours!