Digital Developer Conference: Cloud Security 2021 -- Build the skills to secure your cloud and data Register free

Enabling advanced key usage and management in encrypted container images

High assurance environments such as financial, federal government, and medical industries have strict requirements in terms of protecting the confidentiality of container workloads. To protect these workloads, you can use encryption technologies to secure the confidentiality of workloads and fulfill workload compliance controls (such as those in the NIST Special Publication 800-53).

In previous blog posts and talks, we introduced encrypted container images (ocicrypt) as a way to protect the confidentiality of container workloads, as well as how they can be used in an enterprise setting to help you achieve workload compliance. These provide a good overview of how to protect workloads in a high assurance environment.

In this article, we’d like to dig deeper into a specific aspect of protecting encrypted container images and key management. We will talk about several shortcomings of naive key management and provide a glimpse of the new features in ocicrypt, the keyprovider extensibility that addresses this problem, and an example of a high assurance architecture.

Motivation for the work

When there is encryption, there must be a decryption key. The question of how that key is handled, managed, and distributed is the basis of our discussion.

Let’s start with a standard setup where an enterprise encrypts its workloads. Naturally, it generates a set of encryption/decryption keys to do that. Assuming the company has a good security practice, the encryption/decryption keys are stored in a secure key management server, such as HashiCorp Vault. Only the encryptor of the images (for example, a CI/CD pipeline) and the runtime (for example, Kubernetes, CRI-O, or containerd) have access to the encryption/decryption keys respectively.

This is generally a secure setup and provides moderate auditability of access to keys. However, there is still a significant attack vector that is present in this scenario. In the event of a breach in a cluster, an attacker may be able to gain access to the decryption key and can then use this key to decrypt all the encrypted container workloads. Because decryption of the cryptographic content is done offline, it is difficult to account for a blast radius of a potential breach. Keys are auditable by the key management, but key use is not.

This is not a new problem and the traditional approaches to this issue were mainly key and access rotation, and more logging and monitoring. However, development in key management technology and hardware security components introduced other ways to better handle this problem. This includes performing remote encryption/decryption, allowing every crypto invocation to be audited, and, even better, backing the key by tamper resistant hardware (such as a Hardware Security Module (HSM) or a Trusted Platform Module (TPM)).

Ocicrypt, CRI-O, and containerd

Now let’s relate the previous discussion to the encrypted container images technology, ocicrypt. The encryption schemes that are available in v1.0.x are openpgp, jwe, and pkcs7. These protocols allow you to wrap/unwrap the data encryption keys (DEKs) of the container layer data through a public/private key pair. The libraries that are implemented in Golang require that access to the private key be provisioned on the host.

However, having the worker nodes access the raw key bytes is not ideal, as a node breach would lead to leakage of key material, which can be used to perform unauthorized and un-auditable offline decryption of container workloads. Instead, the ideal scenario is that the container tooling would be able to perform secure wrapping/unwrapping of the DEK without actually reading in the bytes of the key. This prevents any leak of material that allows an offline decryption of other images. There are two mechanisms to do this and one is through remote unwrapping. Examples include the IBM Key Protect and the Azure Key Vault wrapping/unwrapping services. Unfortunately, the state of this protocol is specific to the implementation of the key management server, which means it is difficult to integrate this protocol into encryption protocol standards such as pkcs7 and jwe. An effort was made to standardize this through the OASIS Key Management Interoperability Protocol (KMIP) standard, however, adoption of this standard is still not widespread. The other option is through hardware security protocols such as PKCS11, which is the most mature standard. However, it has still not made its way into the mainstream protocols (although there is an experimental implementation in ocicrypt v1.1.0).

These challenges led to the development of the keyprovider feature of ocicrypt, an extensible way to use custom DEK wrap and unwrap functions in the encrypted container images ecosystem. This allows services and vendors to provide plugin modules or middleware to leverage the advanced key management interfaces they provide, which results in a stronger threat model for keys to protect container workloads.

What is key provider

The key provider protocol is yet another implementation of the ocicrypt key wrapper interface. This protocol allows the ocicrypt library to integrate with any key management service, which can provide key encryption keys (KEK) for container image encryption/decryption. This protocol even supports integration of the ocicrypt library with multiple key management services (KMS). This is achieved by mapping a protocol name to key provider middleware, which can interact with specific KMS for retrieving KEKs. The protocol name, along with the KEK metadata, is defined in an image layer manifest during image encryption. The key provider protocol makes key-unwrapping calls to specific key provider middleware based on the protocol defined in the image manifest. The list of protocol names and the corresponding key provider middleware are defined in the configuration file of the ocicrypt library, the path of which is generally passed over the environment variable OCICRYPT_KEYPROVIDER_CONFIG.

Key provider middleware

Key provider middleware acts as an interface between the ocicrypt key provider protocol and the key management service. This middleware implements wrap and unwrap methods, which generally consist of client calls to KMS for retrieving the KEKs required for wrapping and unwrapping keys.

Architecture diagram

Figure 1 illustrates the end-to-end flow of image encryption and decryption using ocicrypt and its key provider integrated with commonly used tools. Ocicrypt is a crypto library that is used by Skopeo, Buildah, CRI-O, and containerd. The key provider plugin and middleware are introduced to support the key provider protocol. With the key provider plugin built into ocicrypt and external key provider middleware, any key management service can be supported for customer needs.

Figure 1

Figure 1. Ocicrypt key provider end-to-end flow with integration of container image tools.

During the DevOps process, the provided key provider middleware can be called to retrieve an image encryption key from a KMS to encrypt the container image. When a container instance is launched from this encrypted image, the key provider middleware is called to reach out to the KMS to retrieve the decryption key.

Implementation

There are flexible ways to implement key provider middleware:

  1. gRPC. You can implement the middleware in any language that supports gRPC by running a gRPC server and you should serve the wrap and unwrap key calls from the key provider protocol. The middleware can listen over UNIX/TCP sockets. The middleware should implement WrapKey (for image encryption) and UnWrapKey (for image decryption) methods. You can find the protocol buffer in the ocicrypt library.

  2. Binary executable. You can also implement the key provider middleware as a simple binary executable in any language. You should implement methods to wrap and unwrap keys by using KEKs retrieved by making client calls to the key management service.

    In cases of image encryption or decryption, the ocicrypt library should pass a serialized JSON KeyProviderKeyWrapProtocolInput as part of the standard input to the command and a serialized JSON KeyProviderKeyWrapProtocolOutput should return as a standard output of the command.

Following are deserialized STDIN/STDOUT JSON structures of the WrapKey and UnWrapKey methods.

  • Deserialized JSON input to wrap key method:

    KeyProviderKeyWrapProtocolInput {
      "op": "keywrap",
      "keywrapparams": {
        "ec": {
          "keyprovider-1": [
            [ < base64 encoded bytes > ],
            [ < base64 encoded bytes > ],
            ...
          ]
        },
        "optsdata": < base64 encoded bytes >
      }
    }
    
  • Deserialized JSON output of wrap key method:

    KeyProviderKeyWrapProtocolOutput {
      "keywrapresults": {
        "annotation": < base64 encoded bytes >
      }
    }
    
  • Deserialized JSON input to unwrap key method:

    KeyProviderKeyWrapProtocolInput {
      "op": "keywrap"
      "keyunwrapparams": {
        "dc": {}
        "annotation": < base64 encoded bytes >
      }
    }
    
  • Deserialized JSON output of unwrap key method:

    KeyProviderKeyWrapProtocolOutput {
      "keyunwrapresults": {
        "optsdata": < base64 encoded bytes >
      }
    }
    

Building a sample Golang application or binary

There is a sample Golang application that you can build as a binary executable. You can use this application for a sample image encryption where it encrypts the symmetric key, which is passed as standard input (STDIN) from the ocicrypt key provider protocol. The symmetric key is then encrypted with a statically stored KEK in the application and returns the wrapped key along with the key URL and encryption type in an annotation as a standard output (STDOUT) to the key provider protocol. You can use the same application to decrypt the sample encrypted image, where it decrypts the wrapped key in an annotation passed as STDIN using the same statically stored key and returns the image decryption key back to the key provider protocol.

Configure the ${HOME}/ocirypt.conf file as follows:

$ cat /home/vagrant/ocicrypt.conf
  {
      "key-providers": {
          "simplecrypt": {
              "cmd": {
                  "path":"/home/vagrant/simplecrypt",
                  "args": []
              }
          }
      }
  }

Prepare a sample image to encrypt or use an already built image from any public/private registry by pulling it into a local repository. The image should be oci complaint.

$ skopeo copy docker://docker.io/library/alpine:latest oci:alpine
    Getting image source signatures
    Copying blob 05e7bc50f07f done
    Copying config 5c41fd95ee done
    Writing manifest to image destination
    Storing signatures

Encrypt the image as follows:

$ OCICRYPT_KEYPROVIDER_CONFIG=/home/vagrant/ocicrypt.conf bin/skopeo copy --encryption-key provider:simplecrypt:test  oci:alpine oci:alpine-encrypted
    Getting image source signatures
    Copying blob 05e7bc50f07f done
    Copying config 5c41fd95ee done
    Writing manifest to image destination
    Storing signatures

Here is the sample image layer manifest with embedded annotation:

 {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
      "digest": "sha256:fc7c0f56048fc7f09adf5e9fafc70be0a670d710eb1a3d7b031b081aceb4f8c0",
      "size": 376,
      "annotations": {
        "org.opencontainers.image.enc.keys.provider.simplecrypt": "eyJrZXlfdXJsIjoiaHR0cHM6Ly9rZXktcHJvdmlkZXIva2V5LXV1aWQiLCJ3cmFwcGVkX2tleSI6IjdTVzFHWm1XUGV5WGl4MktNd1phUDJoanhwS0NiL3JqVTFESGtFVlExMUZibXJNaW1oditBVncrd2E3OGxhb0N4bWlpVzQ5aFlaUTdzbXZXU3NLWjFqWGVrZGNXZlE3YnhMSVdFR2JyYldLNThhYVZvOUxMOCs0SnhEc3UrVXdMNjJDNUoyUUZjSGpHQm90a2JwQU5pZURKOTFWV3BsKzVSdFAvaEJJNnFJMjZNNkwwd20wNWlIMDZQYzZQNmRUU0RVNkY3T2QzOGhxbkIrdk9WYm9MS2xxeXRPeHlJU2NCWHhSV1JRbWpPL2ZWT29iMG1pclZ5cnNFRW1HRVU1cTgxSkx6WTZDUkRianhET2YxeFdjTkQ5d0czUVpzdWhGcVVBeEVaTDA9Iiwid3JhcF90eXBlIjoiQUVTIn0="
      }
}

Here is the Base64 decoded annotation:

{
"key_url": "https://key-provider/key-uuid",
"wrapped_key": "7SW1GZmWPeyXix2KMwZaP2hjxpKCb/rjU1DHkEVQ11FbmrMimhv+AVw+wa78laoCxmiiW49hYZQ7smvWSsKZ1jXekdcWfQ7bxLIWEGbrbWK58aaVo9LL8+4JxDsu+UwL62C5J2QFcHjGBotkbpANieDJ91VWpl+5RtP/hBI6qI26M6L0wm05iH06Pc6P6dTSDU6F7Od38hqnB+vOVboLKlqytOxyIScBXxRWRQmjO/fVOob0mirVyrsEEmGEU5q81JLzY6CRDbjxDOf1xWcND9wG3QZsuhFqUAxEZL0=",
"wrap_type": "AES"
}

The annotation shows a structure based on an implementation. This metadata is fully customizable and can be used for an example to store key URL information for remote unwrapping or hardware configurations to perform the unwrapping of the key.

Decrypt the image as follows:

$ OCICRYPT_KEYPROVIDER_CONFIG=/home/vagrant/ocicrypt.conf bin/skopeo copy --decryption-key provider:simplecrypt:extra-params oci:alpine-encrypted oci:apline-decrypted

    Getting image source signatures
    Copying blob 4029b2314db9 done
    Copying config 5c41fd95ee done
    Writing manifest to image destination
    Storing signatures

Integration with Intel® Security Libraries for Data Center (Intel® SecL-DC)

Key broker service and platform attestation

Circling back to the original motivation of protecting container workloads in a high assurance environment, let us now explore how you can use the new key provider feature with a technology such as Intel® SecL-DC, that authenticates every decryption/unwrap request and does so not only through regular authentication mechanisms, but also through a full attestation of the requesting system.

To demonstrate the design, we integrated the key provider plug-in model with the Intel® SecL-DC open source project that provides attestation services to support TPM (Trusted Platform Module) based platform attestation, SGX (Secure Guard Extensions) attestation, and TDX (Trusted Domain Execution) attestation, and several use cases based on these attestation services. One of the key use cases is the Intel® SecL-DC key broker service, which manages key policies to restrict key access based on the attestation result. The key policy can be defined when the key is used to request platform, SGX, or TDX attestation. Only successful attestation grants the key request. For example, a key can only be released to the host if the platform is identified as trusted with platform attestation service, or the request comes from a SGX enclave. For the demonstration, we used the platform attestation.

Figure 2 illustrates the integration diagram. The top half is the same encryption and decryption part as Figure 1. The bottom part is the Intel® SecL-DC components, including platform attestation (trust agent and attestation service) and key broker service (with key proxy and backend KMIP server).

Figure 2

Figure 2. Ocicrypt key provider integration with Intel® Security Libraries for key management and attestation (components included in the dotted block located in the bottom half of the diagram).

From a high-level perspective, seven steps are involved (as illustrated in Figure 2):

  1. At first, a platform attestation service is deployed that attests hosts with the trust agent deployed on each host. This provides a foundation to identify which nodes are trusted. The key management service uses the attestation result to determine if the key should be released to the requestor.
  2. The second step is for image encryption. The key provider middleware calls into the key broker service to retrieve the enc key for image encryption.
  3. When it is time to decrypt the image, the key provider middleware on the host side sends a key request to the key broker service over the key proxy.
  4. The key proxy fetches the host trust report from the attestation service before forwarding the key request.
  5. The key proxy sends the key request along with the host attestation report to the key broker service.
  6. The key broker service evaluates if the host is trusted, gets the key from the back-end KMIP server, and wraps the key to a binding key generated from the TPM on the host. If the host is not trusted, the request is denied.
  7. Finally, the key provider unwraps the image decryption key from the TPM on the host and the key is used by ocicrypt to decrypt the image.

Upstreaming

The key provider feature was upstreamed to the master branch of the following projects and is available within the following versions:

  • ocicrypt (>=v1.1.0)
  • containerd (imgcrypt) (>=v1.1.0)
  • CRI-O (master branch, will be enabled in cri-o >=v1.21)
  • Skopeo (master branch)
  • Buildah (>= master (03/05/2021), will be enabled in buildah v1.20)

Conclusion

When dealing with encryption, key management can be a tricky issue to handle, especially in enterprise and high assurance environments. In this article, we outlined several ideas to solve this and introduced the new ocicrypt keyprovider feature that allows for extensible enterprise and advanced key management solutions for securing container image encryption. As an illustration, we used Intel® SecL-DC to describe an end-to-end high assurance architecture including host attestations and remote decryption.

The keyprovider feature is now upstreamed to most of the container ecosystem and we hope to see interesting use cases and implementations from the community for this pluggable interface. From the perspective of a cloud provider, enterprise customer, or security solution for containers, the ocicrypt keyprovider feature will be helpful to meet security and compliance standards today.