Virtualization and containerization significantly benefit efficiency, adaptability, and scalability of workloads. However, workloads might be hosted on an environment sharing a pool of physical platforms in a data center or multi-tenant cloud. Enterprises have security concerns on whether workloads are being run on platforms that are trustworthy, in terms of the integrity of the platform, its locality and metadata, and its ability to establish itself in a root of trust. Another security concern is the confidentiality of the workload image and its key protection, which is especially important when dealing with regulated or sensitive workloads and data.
The foundation of any data center or edge computing security strategy should be securing the platform on which data and workloads will be executed and accessed. The physical platform represents the first layer for any layered security approach and provides the initial protections to help ensure that higher-layer security controls can be trusted.
This article presents an innovative, robust technology solution with policy-based governance to automate the process of mitigating many of the security concerns for containers, which are shown in the following figure.
You can define policies to:
- Ensure workloads in the cluster are only run on trusted physical platforms.
- Report untrusted platforms to a management alert dashboard.
- Encrypt workload images before being uploaded.
- Only decrypt and launch images that are run on trusted platforms with the appropriate metadata requirements (such as location, asset tag, and so on).
In this article, we will start by describing the details of the building blocks of a trusted container platform, and the technologies used, including Intel® Security Libraries for Data Center (ISecL-DC), Red Hat OpenShift, IBM Cloud Pak for MultiCloud Management, and IBM Encrypted OCI Container Images. Then, we will describe the architecture for this trusted container platform. Finally, we demonstrate the features of the components to build the trusted container platform including the implementation in support of the outlined policy.
This project is a collaboration among NIST, IBM, Red Hat, and Intel. The objective of this article is to share some early development of the trusted container platform and to demonstrate a prototype that leverages open-source software components and commercial of-the-shelf technology. This article is the first in a series of articles where we will share the research conducted by the team.
Elements of a trusted container platform
A high-assurance-enabled cloud should be able to secure and govern the use of container workloads to a higher precision. It should be able to tie a workload to a physical entity, and subsequently, high order logical entities. In order to do so, we should leverage these capabilities or technologies:
- Hardware root of trust
- Workload placement and orchestration
- Workload encryption
Each of these capabilities are delivered by open-source software components and commercial off-the-shelf technology.
Hardware root of trust
Hardware-based security techniques can help mitigate threats by establishing and maintaining platform trust—an assurance in the integrity of the underlying platform configuration, including hardware, firmware, and software. By providing this assurance, security administrators can gain a level of visibility and control over where access to sensitive workloads and data is permitted. Platform security technologies that establish platform trust can provide notification or even self-correction of detected integrity failures.
The Intel Security Libraries for Data Center (ISecL-DC) component consists a set of building blocks that discover, attest, and use hardware security features, such as Intel® Trusted Execution Technology (Intel® TXT), Intel® Boot Guard, and Intel® Software Guard Extensions (Intel® SGX), to help enable critical cloud security and confidential computing use-cases. The building blocks in ISecL-DC provide a consistent set of APIs for easy integration with cloud management software and security monitoring and enforcement tools. ISecL-DC provides a middleware that readily integrates platform security features with cloud orchestration and services.
Workload placement and orchestration
Platform information and verified firmware and configuration measurements that are retained within an attestation service can be used for policy enforcement in a variety of use cases. One example is orchestration scheduling. Cloud orchestrators, such as Kubernetes and OpenStack, provide the ability to label server nodes in their database with key value attributes. The attestation services can publish trust and informational attributes to orchestrator databases for use in workload scheduling decisions. In addition, the orchestration system should provide the visibility into the attestation state of the machines.
Two such components can be used to deliver this capability:
Red Hat OpenShift is a CNCF-certified Kubernetes distribution, providing an open hybrid cloud platform with enterprise-class resiliency. Red Hat OpenShift offers automated installation, upgrades, and lifecycle management throughout the container stack—the operating system, Kubernetes and cluster services, and applications—on any cloud. The platform offers developer- and operational-centric tools the ability to enable application development, deployment, scaling, and lifecycle maintenance for long-term innovation. Red Hat OpenShift is focused on security at every level of the container stack and throughout the application lifecycle. Red Hat OpenShift runs on Red Hat Enterprise Linux (RHEL) CoreOS, a container-optimized operating system. IBM Cloud Pak components, which exclusively run on OpenShift Clusters, leverage the security capabilities built into Red Hat OpenShift, and jump start the journey to building a cloud-native business application.
Multicloud management technology in IBM Cloud Pak for Multicloud Management or Red Hat Advanced Cluster Management for Kubernetes enhances the security lifecycle of your hybrid cloud environments. Enterprises must meet internal standards for software engineering, secure engineering, resiliency, security, and regulatory compliance for workloads hosted on hybrid clouds. Teams that provide enterprise cloud platforms, as well as application business units that run their business applications on similar cloud platforms, can use this governance capability to gain visibility and drive remediation for various security and configuration aspects to help meet such enterprise standards. IBM Cloud Pak for Multicloud Management and Red Hat Advanced Cluster Management are two technology offerings that provide similar multicloud management capabilities that can be interchangeable in this prototype. For our article series, we are using IBM Cloud Pak for Multicloud Management.
Consumers who place their workloads in the cloud or on the edge are typically forced to accept that their workloads are secured by their service providers without insight or knowledge as to what security mechanisms are in place. The ability for users to encrypt their workload images can provide at-rest cryptographic isolation to help protect consumer data and intellectual property. When the runtime node service receives the launch request, it can detect that the image is encrypted and make a request to retrieve the decryption key. This request can be passed through an attestation service so that an internal trust evaluation for the platform can be performed. The key request is forwarded to the key broker with proof that the platform has been attested. The key broker can then verify the attested platform report and release the key back to the Cloud Service Provider and node runtime services. At that time, the node runtime can decrypt the image and proceed with the normal workload orchestration. The disk encryption kernel subsystem can provide at-rest encryption for the workload on the platform.
Encrypted Container Images is a capability introduced by IBM Research in container build tools and runtimes such as skopeo, buildah, containerd, and OpenShift (via cri-o ), all of which allow the encryption and decryption of container images. This is based on the OCI container standard and ensures confidentiality of container images as soon as they leave the pipeline, until they are run on a trusted compute node with access to the decryption keys. In an event of a registry compromise, this ensures that the confidentiality of container images stay intact, and we can cryptographically associate trust with images.
Trusted container platform architecture
Let’s see how we can put these technologies together to form an example of a trusted container platform.
Here is an overview of the architecture for a trusted container platform, followed by a description of how the technologies map onto the components.
Overview of trusted container platform components
- ISecL-DC Server (on the left of the architecture diagram) contains the key broker service, attestation service, and utilities for attesting the hardware root of trust and host measurements
- There is a managed cluster (in the middle of the architecture diagram) and a management cluster (on the right of the diagram).
- The managed cluster is the cluster in which our trusted workloads will run. We can imagine a multiplicity of managed clusters that are governed by IBM Cloud Pak for Multicloud Management.
- The management cluster contains the control plane for IBM Cloud Pak for Multicloud Management and DevOps-related tooling.
- For our setup, we have two Red Hat OpenShift clusters: management clusters and managed clusters. The management cluster is enabled on VMWare with VSAN on three bare metal servers. The managed cluster is enabled on Kernel-based Virtual Machine (KVM), but with a hybrid install by placing worker nodes on both a KVM VM Guest and a bare metal server.
Attestation of bare metal nodes
- Each bare metal node has a Trusted Platform Module (TPM) and runs a bootloader up to the operating system stack which has hardware root of trust for measurement using Intel TXT.
- These measurements are collected by the ISecL-DC trust agent on the nodes.
- The node trust status is verified via the ISecL-DC attestation service and then updated by the attestation hub into each Red Hat OpenShift cluster.
- The Key Broker (within ISecL-DC services group in the architecture diagram) manages the keys for all clusters and helps ensure that they can only be accessed by attested trusted nodes. It is designed to require trust attestation in order to release keys.
- Red Hat OpenShift contains a pipeline that will encrypt container images with the help of the skopeo tool.
- The counterpart to decrypting the image is cri-o which the OpenShift container runtime automatically deployed on each worker node.
- The encryption/decryption tools skopeo and cri-o have a plugin to talk to ISecL-DC services in order to perform secure key exchange of the encryption/decryption keys.
- Each managed cluster contains an IBM Cloud Pak for Multicloud Management Klusterlet, which ensures that each managed cluster adheres to the policy in place.
- Policies are created in the IBM Cloud Pak for Multicloud Management Hub, and these policies are propagated to each managed cluster, where they are enforced.
We have 3 policies put in place:
- Ensure all nodes in clusters are trusted
- Ensure user container workloads are encrypted
- Ensure that a DevOps pipeline is in place to enforce the building of applications with an encryption policy
Processes of a trusted container platform
Now that we’ve established the technologies that form the building blocks, let’s see how we used the technologies above to achieve the desired policy we put forward at the start of this article.
Ensure nodes in the container platform are trusted
To help ensure a secure and trusted runtime environment, we want to assert that our clusters only run trusted nodes, rooted in hardware root of trust, which in our case is Intel TXT) and Trusted Platform Module (TPM). Node platform attestation provides the capability in a cluster to know which node is in a trusted or untrusted state so that the cluster can leverage such information to schedule workloads on a trusted node if required and provide mitigation if a node becomes untrusted.
ISecL-DC platform attestation and cluster integration includes three components:
- A trust agent deployed on each node
- A verification service
- An integration hub
Using hardware (Intel TXT or Intel BootGuard) as the core root of trust for measurement to establish the chain of measurement for each component, the trust agent can respond to the verification service for a TPM quote to report the host manifest. The verification service verifies the measurement with a trusted measured database and asserts if the node is trusted or not. The integration can retrieve the node trust status including node asset tag and push them to the orchestration system as labels.
This process helps ensure that the compute nodes that are in the cluster are trusted and asset tags (such as geo-location) are known, and if, for any reason, the integrity of a node is compromised, the node would be shut down and removed from the secure environment. Therefore, all nodes that are running in the cluster should be attested and trusted at all times. The trust status of the nodes is defined as a policy and is enforced through the use of IBM Cloud Pak for Multicloud Management. If a node is detected to lose its trust status from a failure to attest, an event is created in IBM Cloud Pak for Multicloud Management and the node is removed from the cluster.
Demo: Attest compute nodes and govern their trust
In the following video, we demo the ability to attest the compute nodes and show the ability to govern their trust via IBM Cloud Pak for Multicloud Management and Red Hat OpenShift.
Ensure container workloads are encrypted and secure
The process of ensuring that the container workloads are secure is twofold. We first need to ensure that there is a secure process for container workloads. Namely, we need to ensure that there is a DevSecOps process in which we validate the security of our container workload. We use the secure pipelines from Red Hat OpenShift to act as a gate to ensure proper vulnerability management, and add-on to the process by using skopeo and buildah encryption capabilities to encrypt the workloads. During the encryption process, the Key Broker (an ISecL-DC component), is responsible for the creation of the node according to the policy put forward. In this case, an example policy can be that a container workload should only be run on a trusted and attested node, or something more specific, like requiring a certain asset-tag of “region:EU”, which is bound to the hardware TPM as part of ISecL-DC capabilities. Once this is done, the encrypted image is then hosted in a container registry.
Next, to tie things together, we need to ensure that our labeled workloads only run in the secure environment. When the OpenShift container runtime downloads the container image from the registry, it detects that it is encrypted and reaches out to the ISecL-DC Key Broker to obtain the decryption key. At this point, ISecL-DC attests the node and ensures that it is trusted and has the required asset-tags as configured in the policy in the DevSecOps pipeline. Only if this is true, the key will be released, and the container image will be decrypted and run. This acts as an additional layer of protection if the workload is accidentally or maliciously run on an untrusted node.
Demo: Creating encrypted workloads
In the following video, we demo the creation of an encrypted workload via a DevOps pipeline and show that it can only be run on trusted nodes.
Summary and next steps
In this article, we’ve laid out some of the requirements for creating a trusted container platform to support regulated or sensitive workloads. We’ve highlighted some technologies that we’ve used to construct such a prototype platform and shown how they can be used in concert to deliver such a platform.
We are looking for the open source community to help contribute to and advance the trusted container platform. You can contribute by going to our GitHub.
In our next article, we will detail the architecture as well as the steps needed to set up a trusted container platform like we have shown in this article.