A journey to Kubernetes security
Prioritize security when modernizing your apps
In recent years, enterprises and developers have started adopting the idea of application modernization by migrating their apps to cloud, applying containerization, using serverless architecture, and — most importantly — breaking down their monolithic applications into microservices. Kubernetes plays an important role in this adoption because of the advantages it offers: running and coordinating containerized services across a cluster of machines.
But as we developers are moving into this Kubernetes world, there is one thing we need to be careful of: security. Various blogs, videos, books and tutorials educate developers on how to use Kubernetes, but only a few mention the security controls that you need to add to secure your application in the cluster.
Let’s start the security discussion by taking a look at the different attack vectors of the Kubernetes architecture:
|Kubernetes architecture and its attack vectors|
Container image security
Just as developers need to adopt the practice of secure coding for application security, they also need to ensure the security of the image layers built on top of the application. For example, a Java application needs a Java development kit as a base image, which in turn needs a base operating system image. Even if you ensure that your own java application is free of vulnerabilities, if the base images are not secure, the container could be vulnerable to attacks. Take a look at the following diagram to understand the vulnerabilities:
|Container image security|
Safeguarding a Docker image is a lot harder than attacking it, because hackers just need to find one vulnerability to exploit the image and either use that exploitation to get root privileges, to perform remote execution, or to get control of the host machine.
One of the main sources of exploitation of a Docker image is a vulnerable package. For example, a Docker image vulnerability (
CVE-2019-5021) that uses the vulnerable shadow package allows any users or hackers with shell access to elevate their privileges to root within the container. Another vulnerable package is
sqlite3 3.26.0, which has a vulnerability that allows remote code execution if an attacker sends a malicious SQL command.
It is hard to keep to track of all the packages an application uses and to make sure they are not vulnerable. Vulnerability scanners like the Vulnerability Advisor on IBM Cloud can be very resourceful for scanning images you submitted to the registry and notifying you about the vulnerable packages. They tell you how vulnerable packages can be exploited by the attackers and how to resolve the issues.
In addition to vulnerable packages, the Vulnerability Advisor service also tells you if there are any configuration issues that can make your image vulnerable.
You can ensure container image security using various other commercial tools and open-source tools such as Twistlock, CoreOS Clair, Dagda, and Anchore. A good practice is testing your image with more than one tool to cover most of the possible vulnerabilities.
Master and worker node security
Even though most of the cloud providers of Kubernetes services take care of the nodes’ security, you should never fully depend on the cloud provider for security. As shown in the following diagram, you can follow guidelines to help you:
|Master/worker node security|
Pay attention to these good practices:
- Make sure the machine’s operating system meets the Center of Internet Security (CIS) benchmarks.
- Practice the principle of least privilege: do not give root access to anyone unless absolutely required.
- Encrypt the worker node’s file system and make it read-only.
- Use memory encryption to encrypt a part of the memory where the application data is loaded during runtime. This approach is particularly important when you have more than one pods running on a single worker node.
- Encrypt the etcd value store in the master node, because encryption keys are stored there.
API server security
As previously mentioned, the API server controls access to the cluster. So authentication and authorization plays an important role. Implement authentication and access control with the Identity and Access Management (IAM) service provided by most of the cloud providers, or use Kubernetes role-based access control for fine-grained access control based on namespaces, as shown in the following diagram:
|Fine-grained access control|
Traditionally, network segmentation was implemented physically using IP addresses. In the Kubernetes world, where services are running as different containers in a single machine, you can segment a network virtually using namespaces. Allowing or blocking of communication using the IP address is no longer efficient. Instead, the rules are designed based on the service labels or namespaces, as shown in the following diagram:
In the monolithic application, data transfer from one function to another happens in the same application process. With microservices and Kubernetes, the same data transfer happens over the network — which adds additional attack vectors. The service-to-service communication must be certified and encrypted, and Istio is an option to use. Istio is a service mesh that implements certifications and encryption for pod-to-pod communication using mutual Transport Layer Security (TLS), as shown in the following diagram:
|Mutual TLS – Istio|
This article described some of the important security measures for Kubernetes, which you should consider without fail. Security is a priority function when modernizing your apps. The question isn’t if your application is going to be attacked, but when it’s going to be attacked.
For further reading, see the Resources links.
I send a big thanks to Haytham Elkhoja for his valuable feedback on this article.