Encrypt Kubernetes secrets with IBM Cloud Hyper Protect Crypto Services

When it comes to cloud-native applications built with Kubernetes, data protection should cover both Kubernetes secrets and the persistent data stores used by the apps.

A Kubernetes secret is an object that stores sensitive data like a password, a token, or a key. There are built-in secrets that are created automatically by Kubernetes, such as the secret containing credentials to access the API endpoint. There are also user-created secrets, such as storing access information to leverage other IBM Cloud services including Watson and IBM Cloud Container Registry. By default, the Kubernetes master (API server) stores secrets as Base64-encoded plain text in etcd. In order to enable user-managed encryption control for the secrets, IBM Kubernetes Service now provides support for Keep Your Own Key (KYOK) through integration with IBM Cloud Hyper Protect Crypto Services.

The following diagram illustrates the architecture of this integration:

Integration architecture

Users who are looking to safeguard highly sensitive data want to use their own keys for encryption and also require complete control of their encryption keys. For these users, Hyper Protect Crypto Services provides exclusive control over the entire key hierarchy, including the master key of the HSM that protects the secrets. The Level-4 certification assures that the HSM is tamper-proof—it can sense any attempt to compromise the HSM via physical, chemical, or environmental changes and immediately responds by auto-erasing the keys stored, which then invalidates the data that the keys protect.

This tutorial shows you how to encrypt your Kubernetes secrets using IBM Cloud Hyper Protect Crypto Services as the KMS provider. You’ll learn how to create a secret in IBM Cloud Kubernetes, create a root key in Hyper Protect Crypto Services, and encrypt the secrets and etcd component of your Kubernetes master with the root key in your Hyper Protect Crypto Services instance.

Prerequisites

To complete this tutorial, you’ll need:

Estimated time

It should take you about 1 – 2 hours to complete this tutorial.

Steps

To complete this solution, you’ll need to follow these four general steps:

  1. Create a secret in Kubernetes
  2. Create a key in Hyper Protect Crypto Services
  3. Enable KMS encryption in Kubernetes
  4. Verify that the secret is encrypted
1

Create a secret in Kubernetes

Once your IBM Cloud Kubernetes instance is provisioned and you have set up the cluster, you will need to create a secret. A secret stores sensitive data such as a password, a token, or a key where they can be used within your cluster.

Create a secret in Kubernetes

  1. To create a secret in your cluster:

     kubectl create secret generic my-secret --from-literal=username=joeschmoe
    
  2. To verify that your secret was created, run:

     kubectl get secrets
    

    The output should look something like this:

     NAME                TYPE            DATA    AGE
     my-secret           Opaque          1       1m
    
2

Create a key in Hyper Protect Crypto Services

After you create your Hyper Protect Crypto Services instance and initialize the crypto units, you can create a root key that will encrypt your Kubernetes secrets.

Create a root key

To create a root key from the GUI, click Add key and select Create a key. Give your key a name and submit.

Your key should appear in the Keys table.

3

Enable KMS encryption in Kubernetes

Now you can enable encryption for your Kubernetes cluster through the CLI.

Enable KMS encryption in Kubernetes

  1. First, you need the ID of the Hyper Protect Crypto Services instance. To list all your KMS instances, you can run:

     ibmcloud ks kms instance ls
    
  2. You also need the ID for the root key that you just created in your HPCS instance. To list all of the keys under your HPCS instance, run:

     ibmcloud ks kms crk ls --instance-id <HPCS_instance_ID>
    
  3. The last ID you need is the cluster ID. To view details about your cluster, run:

     ibmcloud ks cluster get --cluster <cluster_name>
    
  4. Enable encryption for your Kubernetes service with this command, entering the cluster ID, HPCS instance ID, and root key ID that you retrieved in the previous steps:

     ibmcloud ks kms enable -c <cluster_name> --instance-id <HPCS_instance_ID> --crk <root_key_ID>
    
  5. You can verify that the KMS encryption is enabled with this command:

     ibmcloud ks cluster get -c <cluster_name_or_ID>
    

    Here is the output when the KMS encryption is in progress:

     Name:                   <cluster_name>   
     ID:                     <cluster_ID>   
     ...
     Master Status:          Key Protect feature enablement in progress.
    

    The KMS encryption is enabled when the Master Status displays Ready, for example:

     Name:                   <cluster_name>   
     ID:                     <cluster_ID>   
     ...
     Master Status:          Ready (1 min ago)
    
4

Verify that the secret is encrypted

Now let’s verify whether the secret is encrypted.

Verify that secret is encrypted

  1. To verify that your secret is encrypted, you need to install the etcd command-line tool, etcdctl.

    a. From github, download the latest etcdctl release package for your machine:

     https://github.com/etcd-io/etcd/releases
    

    b. Unzip the etcdctl package, for example:

     unzip ~/Downloads/etcd-v3.4.4-darwin-amd64/etcdctl -d ~/Downloads
    

    c. Move the extracted folder to your binaries directory, for example:

     mv ~/Downloads/etcd-v3.4.4-darwin-amd64/etcdctl /usr/local/bin/etcdctl
    

    d. To verify that etcdctl is installed, run:

     etcdctl --version
    
  2. Set the environment variable to use the correct etcdctl API version:

     export ETCDCTL_API=3
    
  3. To retrieve your secret from etcd, you need to first get the network configuration to reach it.

    a. Download the Calico configuration file that contains information about the etcd for the cluster:

     ibmcloud ks cluster config -c <cluster_name_or_ID> --admin --network
    

    The output should look similar to this:

     The configuration for <cluster_name> was downloaded successfully.
    

    b. Configure the network with the following command:

     /Users/<user>/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/calicoctl.cfg
    

    c. View the configuration file that was downloaded and take note of the etcdEndpoints, etcdCACertFile, etcdKeyFile, and etcdCertFile fields for the next command:

     cat /Users/<user>/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/calicoctl.cfg
    

    The output should be similar to this:

     apiVersion: projectcalico.org/v3
     kind: CalicoAPIConfig
     spec:
     datastoreType: etcdv3
     etcdEndpoints: https://<master_URL>:31773
     etcdKeyFile: '/Users/<user>/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/admin-key.pem'
     etcdCertFile: '/Users/<user>/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/admin.pem'
     etcdCACertFile: '/Users/<user>/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/ca-<location>-<cluster_name>.pem'
    
  4. Next, verify that the secret has been encrypted. Replace the secret_name with your secret that you previously created and the etcdEndpoints, etcdCACertFile, etcdKeyFile, and etcdCertFile fields retrieved in the step above:

     etcdctl get /registry/secrets/default/<secret_name> --endpoints <etcdEndpoints> --key="<etcdKeyFile>" --cert="<etcdCertFile>" --cacert="<etcdCACertFile>"
    

    The output should be similar to this:

     k8s:enc:kms:v1:ibm:...=a?u???T?fE?pC?/?f|???'?z
     ?nI?a,)?
             9??O?{2??]="g?۳o??\5
     ?,a??AW??6Mx??x?5???7       dwX@DG8Dd?԰?ۭ#??[Y?ρF??????a$??9????_ˌ??m??Ɵϭ8?7????????c4L??q1?$0? ??yfzgl?}
                         ??Aynw#?$?J???p?x??pΝ???]ؖE6I?ө?o??t]??p?s?#0%BׇB?????k*֊ؖ??~?B??????V??
    

    You’ll see that the output is unreadable, which means that the secret has been encrypted. All secrets created in the cluster will now be encrypted with the Hyper Protect Crypto Services root key.

    You now have a key that you manage on Hyper Protect Crypto Services that encrypts your secrets in IBM Cloud Kubernetes!

Summary

IBM Cloud Hyper Protect Crypto Services allows for secure key generation and storage, and takes advantage of an industry-leading hardware security module (HSM). This is the only public cloud HSM that offers FIPS 140-2 level 4 data protection, which means that it’s highly tamper-resistant. Store your keys here and you can be sure they’re kept safe from hackers, and even from IBM. No one but you can read them.

The integration between this service and IBM Cloud Kubernetes enables user-managed encryption control for secrets. For more information, check out the following resources: