Advancing container image security with encrypted container images

When it comes to container image security, you might have heard about image signing, such as Docker Content Trust and Red Hat Image Signing. These technologies ensure the integrity and provenance of the container image by verifying that the image was created by someone we trust and that the image has not been tampered with.

This is a great improvement in container image security, but security gaps remain for protecting the confidentiality of the images and ensuring untrusted hosts cannot run them. For example, if a registry is compromised, we don’t want our top secret algorithms to be stolen. On top of that, we want to have additional cryptographic assurance such that if, somehow, an image is stolen, it cannot be run on non-certified machines (for example, from a compliance perspective).

Container Image Security is a problem space that IBM Research has been tackling together with the collaboration of container experts from containerd, Red Hat and the Open Container Initiative (OCI) community. Through this collaboration, an encrypted container image definition around OCI has been developed allowing one to perform encryption of container image layers. Today, encrypted container images are supported in various projects across the ecosystem including containerd, crio, skopeo, and the Docker Distribution project.

Prerequisites

To get the most from this article, you should have basic knowledge in container technology and basic knowledge in cryptography and security.

Estimated time

Take about 10 minutes to read this article.

What are encrypted container images?

Encrypted Container Images are OCI images with encrypted layers. For the purpose of this article, think of layers as portions of a container image. The way to identify if an image includes encrypted contents is simple. If a layer of the image has a mediatype with an +encrypted suffix at the end, the layer is encrypted! For example: application/vnd.oci.image.layer.v1.tar+gzip+encrypted.

This suffix indicates to tools that the portion of the image is encrypted, and the user will provide parameters (via command line arguments or config files) to be able to decrypt the encrypted portion of the image. The types of keys supported should be familiar to developers today (RSA/GPG keys). The underlying protocols used for managing decryption authorization are PKCS#7, JSON Web Encryption, and OpenPGP. These protocols are designed to work with the existing Public Key Infrastructure (PKI) of an enterprise.

Advanced uses of encrypted container images

Through the use of encrypted images, we are able to de-privilege the registry (making the registry open to wider audiences), and we are less worried in the event of a registry compromise. However, protecting content of the encrypted image as it leaves the container build environment to be deployed to registries and then subsequently be used by a container runtime in production is only the tip of the iceberg in terms of how container encryption can be utilized.

Compliance, export control, digital content protection

In high assurance environments, encryption of images can be used to provide guarantees for where an image is allowed to run. Instead of encrypting an image for a developer, “Alice”, encryption can be performed for a “PROD Cluster”, or “PROD Clusters in EU datacenters”. Thus asserting that certain images are only run at particular logical or physical locations. This can be used to enforce policy as well as digital content protection and export control using appropriate key management, trust bootstrapping, and trust attestation.

These concepts are described in more detail in the blog post: How Encrypted Images bring about compliance in Kubernetes.

De-privileging the cloud: TEEs and encrypted images

Another use of encrypted images is in the execution of confidential payloads in Trusted Execution Environments (TEEs). Some examples of these environments are process and virtualization based TEEs such as Intel SGX, AMD SEV, Intel MKTME, and IBM PEF. Using a TEE’s key delivery and attestation services, a container’s image payload is encrypted and payload confidentiality is persisted through encrypted memory during execution of the container. This protects the confidentiality of the container workload even from the cloud provider.

These concepts are described in more detail in the blog post: What You Define is What You Deploy (Encrypted Container Images + TEEs).

Encryption while retaining benefits of containers

One of the key priorities in designing the specification and technology for container image encryption is to ensure that there is little effect on the advantages of containers. For example, one common association with encryption is that it is in direct competition with deduplication. However, in our design, we still allow most deduplication by:

  1. Performing encryption at the layer level, thus allowing sharing of non-sensitive layers
  2. De-associating key authorization with the content of the layer. For example, key authorization to parties are done through the image manifest metadata instead of part of the data blob. This allows sharing of encrypted blobs across multiple recipients

Another cornerstone of container technology is the DevOps pipeline, including scanning of images in the DevOps pipeline. There are multiple suggested ways to have scanning co-exist with encryption:

  1. Perform scanning before pushing to a public registry.
  2. Encrypt only the top-most layers (OS, middleware and packages still scannable).
  3. With a trusted scanner (for example, your own scanning service or trusted scanning provider), one can authorize the scanner’s private key to decrypt the image.

These are some examples of how container image encryption has been designed to work with existing container ecosystems.

Summary

If you’d like to find out more about Encrypted Container Images and how to use them in the context of kubernetes today (in 1.17), swing by our session at Kubecon EU: Where Are Your Images Running? Stop Worrying and Start Encrypting!

In the meantime, to take a deeper dive into Encrypted Container Images and to try out the technology for yourself, several resources are available:

My blog posts:

Past talks:

  • DockerCon 2019: Enabling High Assurance/Sensitive Container Workloads with Encrypted Images – Brandon Lum, Justin Cormack, https://youtu.be/9LyPUy4XYbs
  • Kubecon Shanghai 2019: Protecting Sensitive Code with Encrypted Container Images on Kubernetes – Brandon Lum, Harshal Patil, https://sched.co/NrpO
  • Kubecon Amsterdam 2020: Where Are Your Images Running? Stop Worrying and Start Encrypting! – Brandon Lum, Harshal Patil, https://sched.co/Zepc