Overview

Skill Level: Intermediate

In this recipe, we show how to secure a Kubernetes cluster for running production or enterprise workloads.

Ingredients

To follow and complete this recipe, you need good basic knowledge of Kubernetes, including a firm understanding of the kube-apiserver, etcd, kube-scheduler, kubelet and kube-proxy.

Step-by-step

  1. Kubernetes Security Basic Overview

    The dynamic nature of containers creates several security challenges when running Kubernetes. These challenges are present for any Kubernetes deployment, but especially in medium to large enterprises. See this Kubernetes enterprise guide for a deeper understanding of the requirements Kubernetes should meet in enterprise environments.

    The most common Kubernetes security challenges include:

    • East-West traffic‚ĒĀcontainers are deployed across hosts and sometimes across multiple clouds, which creates large amounts of traffic between hosts or between clouds or data centers, that needs to be monitored and may be intercepted by attackers.
    • Increased attack surface‚ĒĀdepending on the image used to create them, containers have different attack surfaces and vulnerabilities. Docker and Kubernetes themselves can also introduce vulnerabilities that you need to defend against.
    • Security automation is essential‚ĒĀbecause of the fast-moving nature of containers, especially in large enterprise environments, manual security procedures are simply not relevant. You have to automate security and ‚Äúshift left‚ÄĚ to ensure security is built into your process, from initial development through testing and production deployment.¬†

     

    In the next steps, we provide instructions for securing your Kubernetes cluster. Our instructions focus on securing the server components, etc cluster and network communication, not container workloads. 

    These are by no means comprehensive instructions, but they should serve as an initial checklist, which you can build on to meet your organization’s specific security requirements.

  2. Enable TLS

    Enable TLS for all Kubernetes components that support it. Some components enable local ports over HTTP, and you should understand the configuration changes necessary to enforce HTTPS communication. For Kubelets, you can setup TLS bootstrapping, which allows them to create a certificate signing request when they boot.

    For the kube-apiserver, here are the requirements to enable TLS bootstrapping:

    1. Recognize the CA for client certificate
    2. Check that a kubelet belongs to system:bootstrappers group
    3. Authorize kubelet to create a certificate signing request (CSR)

     

    The kube-controller-manager, here are the requirements for TLS bootstrapping:

    1. Enable access to the Kubernetes CA key and certificate you created and distributed
    2. Enabling CSR signing

     

    Requirements at the Kubelet level are as follows:

    1. Set a path to store the key and certificate it generates
    2. Set a path to a kubeconfig file that does not exist, the bootstrapped config file will be placed there
    3. Set a path to a bootstrap kubeconfig file and provide URL for the server and bootstrap credentials
    4. Optional: you can provide instructions to rotate certificates

     

    Here is an example kubeconfig filet that enables TLS bootstrapping:

    apiVersion: v1

    kind: Config

    clusters:

    - cluster:

        certificate-authority: /var/lib/kubernetes/ca.pem

        server: https://my.server.example.com:6443

      name: bootstrap

    contexts:

    - context:

        cluster: bootstrap

        user: kubelet-bootstrap

      name: bootstrap

    current-context: bootstrap

    preferences: {}

    users:

    - name: kubelet-bootstrap

      user:

        token: 07401b.f395accd246ae52d

    For more details see TLS bootstrapping documentation.

  3. Enable RBAC with Least Privilege, Disable ABAC, and Monitor Logs

    Role-based access control (RBAC) is important for enterprise Kubernetes deployments, as it provides granular policy management for access to resources, in particular namespaces. Kubernetes previously offered Attribute-Based Access Control (ABAC) but since version 1.6, it has been replaced by RBAC and should not be used.

    Here is how to enable RBAC on the kube-apiserver:

    --authorization-mode=RBAC

    Here is how to disable ABAC in GKE:

    --no-enable-legacy-authorization

    Keep to the least-privilege principle, and continuously review and improve your RBAC rules to ensure you never grant unnecessary access to a container, service or component. Ensure you store RBAC logs outside your cluster, otherwise the logs represent a security threat in case of compromise.

  4. Use Third-Party Authentication for API Server

    It is strongly recommended to integrate Kubernetes with a third party authentication provider, such as GitHub. This will provide additional security features like multi-factor authentication, and will ensure the kube-apiserver is immutable to users added or removed. If at all possible, ensure users are not managed at the API server level. You can also use an OAuth 2.0 connector such as Dex.

  5. Protect etcd Cluster by Enabling TLS and Setting Up Firewall

    etcd is a sensitive resource that is an attractive target for attackers, because it stores information on cluster state and secrets. Write access to etcd is equivalent to compromise of the cluster, and even read access can be easily used to escalate privileges. 

    To set up TLS for etcd for client-to-server communication, set the following configuration options:

    • cert-file=<path>: Certificate used for SSL/TLS connections to etcd.
    • –key-file=<path>: Key for the certificate (unencrypted).
    • –client-cert-auth: When this is set, etcd checks incoming HTTPS requests for a client certificate signed by the trusted CA
    • –trusted-ca-file=<path>: Trusted certificate authority.
    • –auto-tls: Uses automatically generated self-signed certificates for client connections

     

    To set up TLS for etcd for server-to-server communication, there are similar configuration options:

    • –peer-cert-file=<path>: Certificate used for SSL/TLS connections between peers
    • –peer-key-file=<path>: Key for the certificate (unencrypted)
    • –peer-client-cert-auth: When this is set, etcd check all incoming peer requests for valid, signed client certificates
    • –peer-trusted-ca-file=<path>: Trusted certificate authority.
    • –peer-auto-tls: Uses automatically generated self-signed certificates for peer connections

     

    Learn more about working with peer and client TLS certificates.

    In addition, set up a firewall between your API server and the etcd cluster. Run etcd on separate nodes and set up a firewall on those nodes, for example using Calico.

  6. Rotate Encryption Keys

    In case of a compromise, you need to ensure attackers cannot get very far with the credentials or systems they manage to compromise. A good way to do this is to rotate encryption keys and certificates.

    The Kubernetes client and server certificates are automatically rotated. If disabled, here is how to enable it:

    • On the kubelet process: –rotate-certificates controls if the kubelet will automatically request a new certificate as the expiration of the certificate currently in use approaches. This is a beta feature so you must also enable –feature-gates=RotateKubeletClientCertificate=true
    • On kube-controller-manager: –experimental-cluster-signing-duration controls how long certificates will be issued for.

    The kube-apiserver uses symmetric encryption keys that must be rotated manually, if you do not use a managed service such as GKE or AKS. Here is how to do this:

    1. Add the new encryption key as the second key in encryption configuration file
      1. Restart kube-apiserver processes:
      2. When you deploy kubeadm, the apiserver restarts automatically. Open /etc/kubernetes/manifests/kube-apiserver.yaml and save it to automatically restart. 
        If you have systemd, use systemdctl to restart.
    2. Edit the configuration and move the new key to the first position in the configuration file, meaning it will be used for encryption
    3. Restart kube-apiserver processes again
      Export and import the secrets to encrypt them with the new key using: $ kubectl get secrets -o json | kubectl replace -f –
    4. Edit configuration and remove the old key

Join The Discussion