Learn more >
by Stefan Berger | Published June 25, 2019
Over the past few years, the cloud industry underwent a major shift from deploying monolithic applications in virtual machines to splitting applications into smaller components, called microservices, and packaging them as containers. The successful popularity of containers today is due in large part to Docker containers. Docker is a company that became a primary driving force behind containers: it provided an easy-to-use tool for building and running Docker containers and a Docker container registry for solving the distribution problem.
A major dependency for the success of container technology is securing containers over the various phases of their lifecycle. One security concern with containers is whether individual containers have vulnerabilities within. To identify vulnerabilities, DevOps pipelines for building containers were extended with scanners that search containers for packages with known vulnerabilities and alert their owners or maintainers if one is found. The Vulnerability Advisor on IBM Cloud is one such example.
Another security concern is making sure that the container you are going to run is the one you requested to run and it hasn’t been modified. To address this issue, containers can be protected from modification using digital signatures securely stored in a notary. The Docker Notary is an example for a public repository storing image signatures. Using the notary, a client can verify the signature of a container image to ensure that a container image has not been tampered with since it was signed with the key of its owner or maintainer. A further potential security issue is container isolation. Linux runtime security technologies such as namespaces, cgroups, Linux capabilities, as well as SELinux, AppArmor, and Seccomp profiles, help to confine containers’ processes and isolate containers from each other at run time.
This article addresses a remaining security concern for enterprises about the confidentiality of data and code in container images. The primary goal for container image security is to allow the building and distribution of encrypted container images for making them only available to a set of recipients. While others might be able to access these images, they cannot run them or see the confidential data inside them. Container encryption builds on existing cryptography such as Rivest–Shamir–Adleman (RSA), elliptic curve, and Advanced Encryption Standard (AES) encryption technologies.
This article is based on joint work with Brandon Lum and Harshal Patil.
To get the most from this article, you should be familiar with Linux containers, container images, and have some prior knowledge about security.
Take about 20 minutes to read this article.
There is no existing work in the area of container image encryption that we are aware of. However, there are many existing implementations and products that support data confidentiality and theft protection through encryption on the file system, on the block device level, or on the hardware level. The latter is realized through self-encrypting drives. Also, encrypted virtual machine images exist.
Encrypted file systems exist for many enterprise-ready operating systems and might support mounting encrypted partitions, directories, and might even support booting from an encrypted boot drive. Linux supports encryption on the block device level through the dm-encrypt driver, ecryptfs is one example of an encrypted file system, and there are other open-source file encryption solutions for Linux available. On Windows, the NTFS v3.0 file system supports encryption. Also, many manufacturers build self-encrypting drives. A solution similar to encrypted drives exists for virtual machine images. The open-source QEMU machine (PC) emulator and VMware virtualization products support encrypted virtual machines images.
Data encryption generally addresses data theft protection for a turned-off system. A related technology is the container image signing that the Docker Notary client and server provides. The Docker Notary server operates next to a container image registry. Users of the Docker client tool have the ability to sign a container image and upload the signature to their accounts on the Notary. In this process, a signature is tied to a container image through the path name of the image and its versions. The signature is created over a hash function, which is calculated over a description of all of the image’s contents. This description is known as the container image manifest. The container image signing technology solves the issue of tamper proofing container images and helps to determine container image provenance.
The Docker ecosystem came together to standardize the formats for container images through the Open Container Initiative (OCI) standards group, which now controls the container runtime format (runtime-spec) and container image format (image-spec). Because the team’s work required extensions to the existing container image format, we defined extensions to the standard for supporting encrypted images. The following sections describe the existing container image format and the extensions.
At the top level, a container can consist of a document in Java Script Object Notation (JSON) that represents an image manifest list. For example, you might use this manifest list when multiple architectures or platforms are supported for a container image. The manifest list contains a list of references to container manifests, one for each architecture and operating system combination for the container image. For example, supported architectures include amd64, arm, and ppc64le, and supported operating systems include Linux or Windows. An example of a manifest list is shown in the following screen capture:
The mediaType field describes the exact format of the referenced document. This manifest list allows for future extensions and choosing the appropriate parser for the referenced document.
The level below the manifest list is the manifest. A manifest is also a JSON document and contains an ordered list of references to image layers. These references contain a mediaType describing the format of the layer. The format can describe whether and how the layer is compressed. For example, each layer can be stored as a .tar file containing the files that were added during a particular build step while running 'docker build' on a Dockerfile. For storage efficiency purposes, layers are often also packed using .gzip compressed files. An example of a manifest document is shown in the following screen capture:
As shown, manifests and layers are referenced through a "digest", which is typically a sha256 hash function over their JSON documents. Manifests and layers are usually stored as files in the file system. Typically the file names are the hash functions over the content, making them easy to find and load. A consequence this hash function technique is that a slight modification in a referenced document causes changes to all documents that reference it, chaining all the way up to the manifest list.
For our team’s project, we based image encryption on a hybrid encryption scheme using public and symmetric key encryption. Symmetric keys are used for bulk data encryption (applied to layer encryption), and public keys are used for wrapping symmetric keys. We used three different encryption technologies, based on public keys: OpenPGP, JSON Web Encryption (JWE), and PKCS#7.
OpenPGP is an encryption and signing technology that is typically used for encrypting and signing of email messages. Open source communities also frequently use it for signing source code commits (tags) in git repositories. It is an Internet standard defined by the Internet Engineering Task Force (IETF) in RFC480 and can be regarded as the open version of the preceding proprietary PGP technology.
OpenPGP has its own format for RSA keys. The keys are typically stored in a keyring file and can be imported from plain OpenPGP key files. A particularly user-friendly aspect of the OpenPGP keyring is that public keys can be associated with their owner’s email address. You can deal with multiple recipients of a messages by simply selecting a list of recipients by their email addresses, which then maps into the the public keys for those recipients. In addition, a web of trust was designed around this technology: you can find many users’ public keys organized by their email address. For example, these keys there are often used for signing git tags.
You can use an OpenPGP encrypted message format to encrypt a bulk message for multiple recipients. The header of the OpenPGP message holds one block for each recipient. Each block carries a 64-bit key identifier of the key, which directs the decryption algorithm where to try the decryption with the matching private key. After the encrypted blob inside a block is decrypted, it reveals a symmetric key that can be used for decrypting the bulk message. Each recipient’s public key encrypted blob reveals the same symmetric key.
We used OpenPGP in a similar way, but the encrypted message that it carries is not the layer. Instead, it holds a JSON document that in turn holds the symmetric key used for encrypting and decrypting both the layer and the initialization vector. We call this key the layer encryption key (LEK), which is a form of data encryption key. Using this method has the advantage that we only need one LEK. With that LEK, we encrypt a layer for one or multiple recipients. Each (container image) recipient can have a different type of key, and it doesn’t need to be an OpenPGP key. For example, it could be a plain RSA key. As long as we can use this RSA key to encrypt the LEK, we can accommodate multiple recipients with different types of keys.
JSON Web Encryption, also known as JWE, is another Internet standard of the IETF and is defined in RFC7516. It is a more recent encryption standard than OpenPGP and therefore uses more recent lower-level ciphers that are intended to meet stricter requirements on ciphers.
From a higher level perspective, JWE works like OpenPGP because it also supports a list of recipients and an encrypted bulk message that is encrypted with a symmetric key that each message recipient has access to. Recipients of the JWE message can have different types of keys, such as RSA keys, certain types of elliptic curve keys intended for encryption, and symmetric keys. Because it is a more recent standard, the possibility still exists to extend JWE to support keys in hardware devices, such as Trusted Platform Modules (TPMs) or Hardware Security Modules (HSMs), using interfaces like PKCS#11 or the Key Management and Interoperability Protocol (KMIP). We use JWE in a similar way to OpenPGP, and we use it when recipients have either RSA or elliptic curve keys. In the future we might extend it to support symmetric keys, such as using KMIP inside an HSM.
PKCS#7, also known as Cryptographic Message Syntax (CMS), is defined in the IEFT standard RFC5652. According to Wikipedia, “It can be used to digitally sign, digest, authenticate or encrypt any form of digital data.”
It resembles the two previously described technologies because it allows for multiple recipients and encryption of a bulk message. Therefore, we used it in similar ways as the other technologies – but only for recipients who provide certificates for encryption keys.
To support the previously described encryption technologies, we extended the manifest document with the following information:
For image encryption support, we defined the following keys:
The value (or content) referenced by each key contains one or multiple encrypted messages for the respective encryption technology. Because these messages might be in binary format, they are encoded with base64. An encrypted layer must have at least one such annotation but can have all of them if its recipient has a sufficient number of different key types.
To identify that a layer was encrypted with an LEK, we extended existing mediatypes with the '+enc' suffix as shown in the following examples:
These examples indicate that a layer is archived in a .tar file and encrypted – or both archived in a .tar file and compressed in a .gzip file and encrypted. The following screen capture shows an example of a manifest that references encrypted layers. It also shows an annotations map holding an encrypted JWE message.
For the symmetric encryption with the LEK, our team chose a recent cipher that supports authenticated encryption and builds on top of the AES encryption standard with 128- and 256-bit keys.
We implemented our design in a recent container runtime community project called containerd. Its golang source code is publicly available at github.com/containerd/containerd. The Docker daemon uses containerd for some of its runtime services, and Kubernetes has a plug-in to use containerd directly. Therefore, we expect that our extensions to support encrypted container images will benefit both.
The implementation of the layer encryption using the LEK is on the lowest architectural level of our extensions. One requirement on the implementation was to accommodate large layers with potentially multiple gigabytes while keeping the memory footprint of the process that performs the crypto operation on the layer at only a few megabytes of heap size.
Golang’s support for authenticated encryption algorithms takes a byte array as input and performs the entire encryption (sealing) or decryption (opening) step on the byte array without allowing further byte arrays to be passed and added to the stream. Because this crypto API required loading the whole layer into memory, or inventing some scheme to modify the initialization vector (IV) for each block, we chose not to use the golang authenticated encryption with associated data (AEAD) support. Instead, we used the miscreant golang crypto library, which supports AEAD on streams (chunks) and implements its own scheme of modifying the IV on each block. In our implementation we split up a layer in 1 MB chunks, which we pass one after the other into the miscreant encryption function. This approach allows us to keep the memory footprint low while using an authenticated cipher. On the decryption side, we perform the reverse and pay attention to errors returned by the Open() function, to ensure the cipher blocks were not tampered with.
On the layer above the symmetric crypto, the asymmetric crypto schemes encrypt a layer’s LEK and initialization vector (IV). To support adding or removing crypto schemes, we register each asymmetric crypto implementation. When the API of the asymmetric crypto code is invoked to encrypt a layer, we call the registered crypto handlers one after the other while passing along the public keys of the recipients. After all recipients’ keys are used for encryption, we return to the annotations map with identifiers of the asymmetric crypto algorithms as map-keys, and with values holding the OpenPGP, JWE, and PKCS#7 encoded messages. Each message holds the wrapped LEK and IV. The annotation maps are stored in the manifest document, as shown the previous screen capture.
We also support adding recipients to an already encrypted image. Image authors can add recipients if they are on an image’s recipients list. That recipients list needs the private key required for unwrapping the layers’ LEK and IV. We then wrap the LEK and IV in a new message using the new recipient’s key and append this message to the annotations map.
We used three types of asymmetric encryption schemes for the different types of keys. We use the OpenPGP keys for OpenPGP message encryption. The PKCS#7 implementation that we use requires x.509 certificates for encryption keys. JWE handles all other types of keys, such as plain RSA keys, elliptic curve, and symmetric keys. We built a prototype extension to JWE that allows for crypto operations using keys managed by a KMIP server.
The containerd runtime environment implements a client tool named ctr for interaction with containerd. We extended ctr to enable testing of our changes and to provide access for containerd users. ctr already implements a subcommand that supports operations on images, such as interaction with an image registry via pulling and pushing.
We extended this subcommand with functionality to encrypt images and to enable individual layers of individual architectures to be encrypted with a particular set of keys. This approach enables users to encrypt only those layers that contain confidential data and keep other layers plain. Plain layers enable layer deduplication, but deduplication on encrypted layers is unlikely to work.
Similarly, we allow decrypting individual layers of individual architectures. We added a subcommand layerinfo, which shows the encryption status of each layer and displays the encryption technologies used to encrypt a layer. For OpenPGP, we can also display the IDs of keys necessary for decryption or convert them into their recipients’ email addresses using the keyring.
Additional container image operations that are affected include exporting and importing of container images. We support encrypting layers on export and decrypting on import. Even though we decrypt the layers for creation of the container’s rootfs filesystem, we keep the encrypted layers and its original metadata files, such as its manifests. This approach allows us to export an image in its encrypted form and to perform authorization checks when users want to run a container with an encrypted image.
When a plain (unencrypted) image is pulled from the registry, it is automatically unzipped and untarred so that containers can be immediately created from it. To facilitate this action for encrypted images, we require that a private key is passed to the images pull command that can decrypt the layers before they are unzipped and untarred. If an image is encrypted with multiple keys, multiple keys can be passed to the pull command. Passing multiple private keys is also supported by all other commands involving decryption. After an encrypted image is successfully pulled from the registry, any user with access to containerd can now create a container from the image. To prove that a user is authorized to use the container image, we require that the user provide the private keys used for decrypting the container. We use the keys to check the authorization of the user by verifying that they can be used for decrypting the LEK of each encrypted layer, and if so, we allow running the container.
This section presents a demo of these encryption steps we used with containderd, using ctr on the command line. It shows the encryption and decryption of a container image.
The first step requires the cloning of the git repository with the containerd encrypted container images (ECI) extension. Then we build containerd and start it. To follow these steps, you should be familiar with setting up a golang development environment:
[stefanb@work containerd]$ git clone https://github.com/stefanberger/containerd.git
Cloning into 'containerd'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 44680 (delta 0), reused 1 (delta 0), pack-reused 44677
Receiving objects: 100% (44680/44680), 50.73 MiB | 30.23 MiB/s, done.
Resolving deltas: 100% (27089/27089), done.
[stefanb@work containerd]$ cd containerd/
[stefanb@work containerd]$ git checkout origin/encryption_code_plus_ctr.pr -b encryption_code_plus_ctr.pr
Branch 'encryption_code_plus_ctr.pr' set up to track remote branch 'encryption_code_plus_ctr.pr' from 'origin'.
Switched to a new branch 'encryption_code_plus_ctr.pr'
[stefanb@work containerd]$ make -j5
[stefanb@work containerd]$ sudo bash -c "PATH=$PATH:$PWD/bin ./bin/containerd"
INFO[2019-04-05T16:32:31.754462825-04:00] containerd successfully booted in 0.009261s
Next, we created an RSA key pair using the openssl command-line tool and encrypted a container with it:
[stefanb@work containerd]$ openssl genrsa -out my1stkey.pem 2048
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x010001)
[stefanb@work containerd]$ openssl rsa -in my1stkey.pem -outform PEM -pubout -out my1stpubkey.pem
writing RSA key
[stefanb@work containerd]$ sudo chmod 777 /run/containerd/containerd.sock
[stefanb@work containerd]$ ./bin/ctr images pull --all-platforms docker.io/library/busybox:latest
docker.io/library/busybox:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd: done |++++++++++++++++++++++++++++++++++++++|
unpacking linux/386 sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd...
unpacking linux/ppc64le sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd...
unpacking linux/s390x sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd...
[stefanb@work containerd]$ ./bin/ctr images encrypt --recipient my1stpubkey.pem docker.io/library/busybox:latest docker.io/library/busybox.enc:latest
Encrypting docker.io/library/busybox:latest to docker.io/library/busybox.enc:latest
[stefanb@work containerd]$ ./bin/ctr images layerinfo docker.io/library/busybox.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:614d50fd0251bb303cbc583db86dbf1ccc40cb734c2c1b16ddddf9ef10ef1825 linux/amd64 755857 jwe [jwe]
0 sha256:9d3f26b310096b4b16597eb7eadf44861f215c4f88654211b3e020b513518105 linux/arm/v5 736777 jwe [jwe]
0 sha256:fa8b697a18f724a0ebc7f1d1ab8ce47a2ee06d61133ec81451ddcb185df0276a linux/arm/v6 902593 jwe [jwe]
0 sha256:dac520950ed1b8963c49f4ad2d3ea25157e989837691878d9f238900018bbe8e linux/arm/v7 704186 jwe [jwe]
0 sha256:6cde893b53eb1834e442d4b1f99a6202ecf952eef98bb98f54ce3e5c99e066fc linux/arm64/v8 796296 jwe [jwe]
0 sha256:bcd9681684ecfbcd2961e6233a8a99f2908d7de970f15b432ffef023749941b3 linux/386 717382 jwe [jwe]
0 sha256:1197b037b1df009b707fd2858f6630319053083a990212db41763e9d25478f75 linux/ppc64le 2182326 jwe [jwe]
0 sha256:4983197819f31f410b7e9ef76d99e691db40bcd23a2a31b4f1b418b2c8e351e1 linux/s390x 2151743 jwe [jwe]
The layerinfo subcommand shows that the busybox image is available for several different architectures and that each architecture has one layer. All layers were encrypted with JWE using the RSA public key.
This last step demonstrates image decryption using the private RSA key. We used the layerinfo subcommand to show that the layer of each architecture has now been decrypted:
[stefanb@work containerd]$ ./bin/ctr images decrypt --key my1stkey.pem docker.io/library/busybox.enc:latest docker.io/library/busybox.dec:latest
Decrypting docker.io/library/busybox.enc:latest to docker.io/library/busybox.dec:latest
[stefanb@work containerd]$ ./bin/ctr images layerinfo docker.io/library/busybox.dec:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985 linux/amd64 755841
0 sha256:c83038a50f6e0d7181947b4991cf3993435db7e3462c0bd13c3a4ae97d6b432c linux/arm/v5 736761
0 sha256:ff0ca67c9bda32fa3a301324fb4c7bd54430e981a0adcf219559a2a3c73fe713 linux/arm/v6 902577
0 sha256:2b35d97f9c8117d50d5d2c9164acf00aa03e41d0a80f9bc2b2044e3e92fa9688 linux/arm/v7 704170
0 sha256:b04ab0589b9a6d0d597a66bae318d4b08520957d4acfc7bf75496e38d3d7c8d3 linux/arm64/v8 796280
0 sha256:79e848d156eaf50a600bb6129f0ee47b2fa6280d25a52d99d7ee48445f186103 linux/386 717366
0 sha256:628fa7149e26dcaa64b2ae1ece67309565d6f7e0a04b97f5813998b196226d80 linux/ppc64le 2182278
0 sha256:681c8c6f047294a46fd0f6a2da748397409e8e4179d307d47f32827a90b0275a linux/s390x 2151695
This example shows that now none of the layers are encrypted.
Container image encryption is an exciting new addition to container image security, addressing data confidentiality and integrity of container images at rest. The technology is built on commonly available RSA, elliptic curve, and AES encryption technologies. It applies the keys to higher level encryption schemes such as OpenPGP, JWE, and PKCS#7. Image authors who are familiar with OpenPGP can encrypt container images to OpenPGP recipients using their email addresses while plain RSA and elliptic curve keys are used by encryption such as JWE.
In the future, our team intends to extend encrypted container images support to make use of hardware crypto devices, which can be accessed using the key management interoperability protocol (KMIP), for example. In the meantime, check out containerd as you work with encrypted container images in your own environment.
As we move into a Kubernetes world, be mindful of security. Learn steps to secure your application in a cluster.
This learning path is comprised of basic to advanced Kubernetes skills.
May 20, 2019
Back to top