Encrypted container images for container image security at rest

Over the past few years, the cloud industry underwent a major shift from deploying monolithic applications in virtual machines to splitting applications into smaller components, called microservices, and packaging them as containers. The successful popularity of containers today is due in large part to Docker containers. Docker is a company that became a primary driving force behind containers: it provided an easy-to-use tool for building and running Docker containers and a Docker container registry for solving the distribution problem.

A major dependency for the success of container technology is securing containers over the various phases of their lifecycle. One security concern with containers is whether individual containers have vulnerabilities within. To identify vulnerabilities, DevOps pipelines for building containers were extended with scanners that search containers for packages with known vulnerabilities and alert their owners or maintainers if one is found. The Vulnerability Advisor on IBM Cloud is one such example.

Another security concern is making sure that the container you are going to run is the one you requested to run and that it hasn’t been modified. To address this issue, you can use digital signatures securely stored in a notary to protect containers from modification. The Docker Notary is an example for a public repository storing image signatures. Using the notary, a client can verify the signature of a container image to ensure that a container image has not been tampered with since it was signed with the key of its owner or maintainer. A further potential security issue is container isolation. Linux runtime security technologies such as namespaces, cgroups, Linux capabilities, as well as SELinux, AppArmor, and Seccomp profiles, help to confine containers’ processes and isolate containers from each other at run time.

This article addresses a remaining security concern for enterprises about the confidentiality of data and code in container images. The primary goal for container image security is to allow the building and distribution of encrypted container images for making them only available to a set of recipients. While others might be able to access these images, they cannot run them or see the confidential data inside them. Container encryption builds on existing cryptography such as Rivest–Shamir–Adleman (RSA), elliptic curve, and Advanced Encryption Standard (AES) encryption technologies.

This article is based on joint work with Harshal Patil.

Prerequisites

To get the most from this article, you should be familiar with Linux containers and container images, and have some prior knowledge about security.

Estimated time

Take about 20 minutes to read this article.

There is no existing work in the area of container image encryption that we are aware of. However, there are many existing implementations and products that support data confidentiality and theft protection through encryption on the file system, on the block device level, or on the hardware level. The latter is realized through self-encrypting drives. Also, encrypted virtual machine images exist.

Encrypted file systems exist for many enterprise-ready operating systems and might support mounting encrypted partitions, directories. Encrypted file systems might even support booting from an encrypted boot drive. Linux supports encryption on the block device level through the dm-encrypt driver, ecryptfs is one example of an encrypted file system, and there are other open-source file encryption solutions for Linux available. On Windows, the NTFS v3.0 file system supports encryption. Also, many manufacturers build self-encrypting drives. A solution similar to encrypted drives exists for virtual machine images. The open-source QEMU machine (PC) emulator and VMware virtualization products support encrypted virtual machines images.

Data encryption generally addresses data theft protection for a turned-off system. A related technology is the container image signing that the Docker Notary client and server provides. The Docker Notary server operates next to a container image registry. Users of the Docker client tool have the ability to sign a container image and upload the signature to their accounts on the Notary. In this process, a signature is tied to a container image through the path name of the image and its versions. The signature is created over a hash function, which is calculated over a description of all of the image’s contents. This description is known as the container image manifest. The container image signing technology solves the issue of tamper proofing container images and helps to determine container image provenance.

Design

The Docker ecosystem came together to standardize the formats for container images through the Open Container Initiative (OCI) standards group, which now controls the container runtime format (runtime-spec) and container image format (image-spec). Because the team’s work required extensions to the existing container image format, we defined extensions to the standard for supporting encrypted images. The following sections describe the existing container image format and the extensions.

At the top level, a container can consist of a document in Java Script Object Notation (JSON) that represents an image manifest list. For example, you might use this manifest list when multiple architectures or platforms are supported for a container image. The manifest list contains a list of references to container manifests, one for each architecture and operating system combination for the container image. For example, supported architectures include amd64, arm, and ppc64le, and supported operating systems include Linux or Windows. An example of a manifest list is shown in the following screen capture:

example manifest list document

The mediaType field describes the exact format of the referenced document. This manifest list allows for future extensions and choosing the appropriate parser for the referenced document.

The level below the manifest list is the manifest. A manifest is also a JSON document and contains an ordered list of references to image layers. These references contain a mediaType describing the format of the layer. The format can describe whether and how the layer is compressed. For example, each layer can be stored as a .tar file containing the files that were added during a particular build step while running 'docker build' on a Dockerfile. For storage efficiency purposes, layers are often also packed using .gzip compressed files. An example of a manifest document is shown in the following screen capture:

example manifest document

As shown, manifests and layers are referenced through a "digest", which is typically a sha256 hash function over their JSON documents. Manifests and layers are usually stored as files in the file system. Typically the file names are the hash functions over the content, making them easy to find and load. A consequence this hash function technique is that a slight modification in a referenced document causes changes to all documents that reference it, chaining all the way up to the manifest list.

For our team’s project, we based image encryption on a hybrid encryption scheme using public and symmetric key encryption. Symmetric keys are used for bulk data encryption (applied to layer encryption), and public keys are used for wrapping symmetric keys. We used three different encryption technologies, based on public keys: OpenPGP, JSON Web Encryption (JWE), and PKCS#7.

OpenPGP

OpenPGP is an encryption and signing technology that is typically used for encrypting and signing of email messages. Open source communities also frequently use it for signing source code commits (tags) in git repositories. It is an Internet standard defined by the Internet Engineering Task Force (IETF) in RFC480 and can be regarded as the open version of the preceding proprietary PGP technology.

OpenPGP has its own format for RSA keys. The keys are typically stored in a keyring file and can be imported from plain OpenPGP key files. A particularly user-friendly aspect of the OpenPGP keyring is that public keys can be associated with their owner’s email address. You can deal with multiple recipients of a messages by simply selecting a list of recipients by their email addresses, which then maps into the the public keys for those recipients. In addition, a web of trust was designed around this technology: you can find many users’ public keys organized by their email address. For example, these keys there are often used for signing git tags.

You can use an OpenPGP encrypted message format to encrypt a bulk message for multiple recipients. The header of the OpenPGP message holds one block for each recipient. Each block carries a 64-bit key identifier of the key, which directs the decryption algorithm where to try the decryption with the matching private key. After the encrypted blob inside a block is decrypted, it reveals a symmetric key that can be used for decrypting the bulk message. Each recipient’s public key encrypted blob reveals the same symmetric key.

We used OpenPGP in a similar way, but the encrypted message that it carries is not the layer. Instead, it holds a JSON document that in turn holds the symmetric key used for encrypting and decrypting both the layer and the initialization vector. We call this key the layer encryption key (LEK), which is a form of data encryption key. Using this method has the advantage that we only need one LEK. With that LEK, we encrypt a layer for one or multiple recipients. Each (container image) recipient can have a different type of key, and it doesn’t need to be an OpenPGP key. For example, it could be a plain RSA key. As long as we can use this RSA key to encrypt the LEK, we can accommodate multiple recipients with different types of keys.

JSON Web Encryption (JWE)

JSON Web Encryption, also known as JWE, is another Internet standard of the IETF and is defined in RFC7516. It is a more recent encryption standard than OpenPGP and therefore uses more recent lower-level ciphers that are intended to meet stricter requirements on ciphers.

From a higher level perspective, JWE works like OpenPGP because it also supports a list of recipients and an encrypted bulk message that is encrypted with a symmetric key that each message recipient has access to. Recipients of the JWE message can have different types of keys, such as RSA keys, certain types of elliptic curve keys intended for encryption, and symmetric keys. Because it is a more recent standard, the possibility still exists to extend JWE to support keys in hardware devices, such as Trusted Platform Modules (TPMs) or Hardware Security Modules (HSMs), using interfaces like PKCS#11 or the Key Management and Interoperability Protocol (KMIP). We use JWE in a similar way to OpenPGP, and we use it when recipients have either RSA or elliptic curve keys. In the future we might extend it to support symmetric keys, such as using KMIP inside an HSM.

PKCS#7

PKCS#7, also known as Cryptographic Message Syntax (CMS), is defined in the IEFT standard RFC5652. According to the Wikipedia entry for CMS, “It can be used to digitally sign, digest, authenticate or encrypt any form of digital data.”

It resembles the two previously described technologies because it allows for multiple recipients and encryption of a bulk message. Therefore, we used it in similar ways as the other technologies – but only for recipients who provide certificates for encryption keys.

To support the previously described encryption technologies, we extended the manifest document with the following information:

  • The OpenPGP, JWE, and PKCS#7 messages are stored in an annotations map that is part of the manifest.
  • One of these maps exists in each referenced layer. The annotations map is basically a dictionary with strings as keys and strings as values (key value pairs).

For image encryption support, we defined the following keys:

  • org.opencontainers.image.enc.keys.openpgp
  • org.opencontainers.image.enc.keys.jwe
  • org.opencontainers.image.enc.keys.pkcs7

The value (or content) referenced by each key contains one or multiple encrypted messages for the respective encryption technology. Because these messages might be in binary format, they are encoded with base64. An encrypted layer must have at least one such annotation but can have all of them if its recipient has a sufficient number of different key types.

To identify that a layer was encrypted with an LEK, we extended existing mediatypes with the '+encrypted' suffix as shown in the following examples:

  • application/vnd.docker.image.rootfs.diff.tar+encrypted
  • application/vnd.docker.image.rootfs.diff.tar.gzip+encrypted

These examples indicate that a layer is archived in a .tar file and encrypted – or both archived in a .tar file and compressed in a .gzip file and encrypted. The following screen capture shows an example of a manifest that references encrypted layers. It also shows an annotations map holding an encrypted JWE message.

an image manifest referencing the encrypted layer

Layer encryption using symmetric keys

For the symmetric encryption with the LEK, our team chose a recent cipher that supports authenticated encryption and builds on top of the AES encryption standard with 128- and 256-bit keys.

An implementation example: containerd

We implemented our design in a recent container runtime community project called containerd. Its golang source code is publicly available at github.com/containerd/containerd. The Docker daemon uses containerd for some of its runtime services, and Kubernetes has a plug-in to use containerd directly. Therefore, we expect that our extensions to support encrypted container images will benefit both.

The implementation of the layer encryption using the LEK is on the lowest architectural level of our extensions. One requirement on the implementation was to accommodate large layers with potentially multiple gigabytes while keeping the memory footprint of the process that performs the crypto operation on the layer at only a few megabytes of heap size.

Golang’s support for authenticated encryption algorithms takes a byte array as input and performs the entire encryption (sealing) or decryption (opening) step on the byte array without allowing further byte arrays to be passed and added to the stream. Because this crypto API required loading the whole layer into memory, or inventing some scheme to modify the initialization vector (IV) for each block, we chose not to use the golang authenticated encryption with associated data (AEAD) support. Instead, we used the miscreant golang crypto library, which supports AEAD on streams (chunks) and implements its own scheme of modifying the IV on each block. In our implementation we split up a layer in 1 MB chunks, which we pass one after the other into the miscreant encryption function. This approach allows us to keep the memory footprint low while using an authenticated cipher. On the decryption side, we perform the reverse and pay attention to errors returned by the Open() function, to ensure the cipher blocks were not tampered with.

On the layer above the symmetric crypto, the asymmetric crypto schemes encrypt a layer’s LEK and initialization vector (IV). To support adding or removing crypto schemes, we register each asymmetric crypto implementation. When the API of the asymmetric crypto code is invoked to encrypt a layer, we call the registered crypto handlers one after the other while passing along the public keys of the recipients. After all recipients’ keys are used for encryption, we return to the annotations map with identifiers of the asymmetric crypto algorithms as map-keys, and with values holding the OpenPGP, JWE, and PKCS#7 encoded messages. Each message holds the wrapped LEK and IV. The annotation maps are stored in the manifest document, as shown the previous screen capture.

We also support adding recipients to an already encrypted image. Image authors can add recipients if they are on an image’s recipients list. That recipients list needs the private key required for unwrapping the layers’ LEK and IV. We then wrap the LEK and IV in a new message using the new recipient’s key and append this message to the annotations map.

We used three types of asymmetric encryption schemes for the different types of keys. We use the OpenPGP keys for OpenPGP message encryption. The PKCS#7 implementation that we use requires x.509 certificates for encryption keys. JWE handles all other types of keys, such as plain RSA keys, elliptic curve, and symmetric keys. We built a prototype extension to JWE that allows for crypto operations using keys managed by a KMIP server.

The containerd runtime environment implements a client tool named ctr for interaction with containerd. We extended ctr to enable testing of our changes and to provide access for containerd users. ctr already implements a subcommand that supports operations on images, such as interaction with an image registry via pulling and pushing.

We extended this subcommand with functionality to encrypt images and to enable individual layers of individual architectures to be encrypted with a particular set of keys. This approach enables users to encrypt only those layers that contain confidential data and keep other layers plain. Plain layers enable layer deduplication, but deduplication on encrypted layers is unlikely to work.

Similarly, we allow decrypting individual layers of individual architectures. We added a subcommand layerinfo, which shows the encryption status of each layer and displays the encryption technologies used to encrypt a layer. For OpenPGP, we can also display the IDs of keys necessary for decryption or convert them into their recipients’ email addresses using the keyring.

Additional container image operations that are affected include exporting and importing of container images. We support encrypting layers on export and decrypting on import. Even though we decrypt the layers for creation of the container’s rootfs filesystem, we keep the encrypted layers and its original metadata files, such as its manifests. This approach allows us to export an image in its encrypted form and to perform authorization checks when users want to run a container with an encrypted image.

When a plain (unencrypted) image is pulled from the registry, it is automatically unzipped and untarred so that containers can be immediately created from it. To facilitate this action for encrypted images, we require that a private key is passed to the images pull command that can decrypt the layers before they are unzipped and untarred. If an image is encrypted with multiple keys, multiple keys can be passed to the pull command. Passing multiple private keys is also supported by all other commands involving decryption. After an encrypted image is successfully pulled from the registry, any user with access to containerd can now create a container from the image. To prove that a user is authorized to use the container image, we require that the user provide the private keys used for decrypting the container. We use the keys to check the authorization of the user by verifying that they can be used for decrypting the LEK of each encrypted layer, and if so, we allow running the container.

Walk through a demo with containerd

This section presents a demo of these encryption steps we used with containderd, using ctr on the command line. It shows the encryption and decryption of a container image.

The first step requires the cloning of the git repository containerd/imgcrypt, which is the sub-project which contains the container image encryption/decryption capabilities. Then you build containerd and start it. To follow these steps, you should be familiar with setting up a golang development environment:

imgcrypt requires containerd 1.3 or later.

Build and install imgcrypt:

# make
# sudo make install

Start containerd with a configuration file that looks like the following example. To avoid interference with a containerd from a Docker installation, use /tmp for directories. Also, build containerd 1.3 from the source but do not install it.

# cat config.toml
disable_plugins = ["cri"]
root = "/tmp/var/lib/containerd"
state = "/tmp/run/containerd"
[grpc]
  address = "/tmp/run/containerd/containerd.sock"
  uid = 0
  gid = 0
[stream_processors]
    [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
        returns = "application/vnd.oci.image.layer.v1.tar+gzip"
        path = "/usr/local/bin/ctd-decoder"
    [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
        accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
        returns = "application/vnd.oci.image.layer.v1.tar"
        path = "/usr/local/bin/ctd-decoder"

# sudo ~/src/github.com/containerd/containerd/bin/containerd -c config.toml

Create an RSA key pair using the openssl command-line tool and encrypt an image:

# openssl genrsa --out mykey.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...............................................+++++
............................+++++
e is 65537 (0x010001)
# openssl rsa -in mykey.pem -pubout -out mypubkey.pem
writing RSA key
# sudo chmod 0666 /tmp/run/containerd/containerd.sock
# CTR="/usr/local/bin/ctr-enc -a /tmp/run/containerd/containerd.sock"
# $CTR images pull --all-platforms docker.io/library/bash:latest
[...]
# $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
   #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
   0   sha256:9d48c3bd43c520dc2784e868a780e976b207cbf493eaff8c6596eb871cbd9609   linux/amd64   2789669                          
   1   sha256:7dd01fd971d4ec7058c5636a505327b24e5fc8bd7f62816a9d518472bd9b15c0   linux/amd64   3174665                          
   2   sha256:691cfbca522787898c8b37f063dd20e5524e7d103e1a3b298bd2e2b8da54faf5   linux/amd64       340                          
# $CTR images encrypt --recipient jwe:mypubkey.pem --platform linux/amd64 docker.io/library/bash:latest bash.enc:latest
Encrypting docker.io/library/bash:latest to bash.enc:latest
$ $CTR images layerinfo --platform linux/amd64 bash.enc:latest
   #                                                                    DIGEST      PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
   0   sha256:360be141b01f69b25427a9085b36ba8ad7d7a335449013fa6b32c1ecb894ab5b   linux/amd64   2789669          jwe        [jwe]
   1   sha256:ac601e66cdd275ee0e10afead03a2722e153a60982122d2d369880ea54fe82f8   linux/amd64   3174665          jwe        [jwe]
   2   sha256:41e47064fd00424e328915ad2f7f716bd86ea2d0d8315edaf33ecaa6a2464530   linux/amd64       340          jwe        [jwe]

Start a local image registry so you can push the encrypted image to it. A recent versions of the registry is required to accept encrypted container images.

# docker pull registry:latest
# docker run -d -p 5000:5000 --restart=always --name registry registry

Push the encrypted image to the local registry, pull it using ctr-enc, and then run the image:

# $CTR images tag bash.enc:latest localhost:5000/bash.enc:latest
# $CTR images push localhost:5000/bash.enc:latest
# $CTR images rm localhost:5000/bash.enc:latest bash.enc:latest
# $CTR images pull localhost:5000/bash.enc:latest
# sudo $CTR run --rm localhost:5000/bash.enc:latest test echo 'Hello World!'
ctr: you are not authorized to use this image: missing private key needed for decryption
# sudo $CTR run --rm --key mykey.pem localhost:5000/bash.enc:latest test echo 'Hello World!'
Hello World!

Summary

Container image encryption is an exciting new addition to container image security, addressing data confidentiality and integrity of container images at rest. The technology is built on commonly available RSA, elliptic curve, and AES encryption technologies. It applies the keys to higher level encryption schemes such as OpenPGP, JWE, and PKCS#7. Image authors who are familiar with OpenPGP can encrypt container images to OpenPGP recipients using their email addresses while plain RSA and elliptic curve keys are used by encryption such as JWE.

In the future, our team intends to extend encrypted container images support to make use of hardware crypto devices, which can be accessed using the key management interoperability protocol (KMIP), for example. In the meantime, check out containerd as you work with encrypted container images in your own environment.

Stefan Berger
Brandon Lum