2021 Call for Code Awards: Live from New York, with SNL’s Colin Jost! Learn more

Protecting data using secret management with Trusted Service Identity

This article explores the problems with current methods of storing secrets in a Kubernetes cluster and introduces you to an open source project called Trusted Service Identity that addresses this problem. This project, recently open sourced by IBM Research, ties secret management technologies with workload identity via host provenance and integrity. For developers who like to play with new technologies, we provide a step-by-step installation and demo.

What is the problem with secrets?

Today, if a microservice needs to retrieve or exchange data with some other external or internal service, it must authenticate itself. This authentication requires secrets (password, API key, certificate, etc.) that are typically stored as a Kubernetes secret.

Even though Kubernetes secrets are designed to be stored securely (for example, encrypted at rest) by the orchestration, they can be simply mounted to an arbitrary container and read by anyone who has access to the namespace, including cloud operators. Those knowing the secret can also access the sensitive data that need to be protected.

The problem is that these operators or admins might not be certified to access this data, or they may lack security clearance or legal requirements like specific citizenship. On top of that, there is an increasing need for an accurate audit trail, to track who (and what process) accessed what data and when.

Typical customer implementation

Let’s take a closer look at a typical customer implementation today. An application is deployed in the Kubernetes cluster and needs to access a database like IBM Cloudant. In order to authenticate itself to this database, an application must present some authentication token or a user ID and password, API key, or certificate. Since this token should not be hard-coded into the image used for instantiating this application, the container must be able to obtain this information during runtime or initialization. Today, this is typically done via Kubernetes secrets.


The problem with this solution is that the secrets are static and long-lived. They are static, because the same secret is given to all instances of the application and that makes the audit much harder. And since the secrets are long lived, there is a higher risk of exposure for a rogue credential usage. To make matters worse, anyone with access to this namespace will be able to read it.

To make it a little more secure, you could move the secret into a secure Vault. All you would keep in the cluster is a secret allowing you to read from Vault, but this will not stop a malicious administrator from eventually gaining access to the sensitive data.

view of secrets

With this model, the secret remain as-is during the deployment, and there is no easy way to control the lifetime of the secret, its expiration, or rotation. In addition, all the secrets are created and managed by humans.

If the deployment spans multiple locations or regions, the management and governance over who has access to what becomes a nightmare. The controls are not unified and tracking it without global visibility becomes very difficult.

Enter Trusted Service Identity by IBM Research

The IBM Research team worked with members of the IBM Cloud and Cognitive Software group to address this problem. The solution is called Trusted Service Identity (TSI). Let’s look at how TSI works.

Every container that requires a secret for accessing sensitive data gets assigned a short-lived measured identity, in the form of a JSON Web Token (JWT) token, that is signed with the root of trust. This measured identity is a form of a digital biometric that contains runtime measurements of the application, like image name, namespace, cluster name, location, data center, and other identifying data.

The actual secret that application needs is stored securely in Vault and is protected by policies that define what application can retrieve it. The Vault auth plugin validates the token and checks the policy. Then, Trusted Service Identity injects this secret to the running container, the same way as regular Kubernetes secrets are done, but without being stored anywhere in Kubernetes. The release of secrets is governed by fine-grained access control policies managed by the Chief Information Security Officer (CISO).

Policies, as well as the secrets, can be dynamically modified and keys revoked or updated, without changing anything in deployed applications, while Trusted Service Identity retains a detailed audit trail of what process accessed the secrets and when.

Use cases for employing Trusted Service Identity

TSI addresses several different use cases for keeping secrets safe. Here are a few interesting examples of when to use TSI:

  • Permissions can be enforced based on the workload identity or a geographical location. For example, some applications must adhere to GDPR standards, like the following one: “Data about Europeans can be only accessed within European geographical boundaries, by an application that was well tested, validated, approved and continuously monitored for compliance”
  • Selected images must be executed only in specific data centers under specific accounts and clusters
  • Provide short-lived access to data, using quick expiration dates
  • Automate the frequent key rotation without modifying the application
  • Common deployment code across multi-clusters, multi-geographical distribution with a location-based data restriction
  • Different data feeds for different environments (development, staging, production)
  • Fine controlled, the dynamic release of secrets and configurations at runtime or specific time

Demo of Trusted Service identity

The following video shows you a quick demo of Trusted Service Identity running on Minikube. If you want to follow the installation and demo instructions, please visit our GitHub project.

Detailed Trusted Service Identity architecture

Architecture diagram simplified

In the example diagram, you can see the basic layout of the architecture before any operations are executed. The sensitive data is stored in IBM Cloudant, so you need to protect access to the database. You have a Kubernetes cluster, extended with TSI, where each node gets intermediate certificate authority (CA), signed by the root CA during the secure bootstrapping of the cluster. Each node has also a JWT Signing Service that contains singing authority. This could be either software or hardware Trusted Platform Module ( TPM).

Root CA is securely stored in Vault, and Vault is extended with the Trusted Service Identity Authentication Vault plugin.

Let’s go through a sample process of steps needed to distribute secrets to the application.

  1. The Chief Information Security Officer (CISO) or some other secure process deploys secret keys in Vault that can access the Cloudant DB. The CISO also creates policies that define the measurements of the application that can access these secrets. These measurements represent the identity of the application.
  2. The developer deploys an application in this cluster. This application wants to access the sensitive data stored in IBM Cloudant. The cluster operator annotates the deployment of this application with the names and types of the secrets to be injected and their location.

           admission.trusted.identity/inject: "true"
           tsi.secrets: |
               - tsi.secret/name: "mysecret1"
                 tsi.secret/role: "demo"
                 tsi.secret/vault-path: "secret/ti-demo-all"
                 tsi.secret/local-name: "mysecrets/myubuntu-mysecret1"
               - tsi.secret/name: "mysecret2"
                 tsi.secret/role: "demo"
                 tsi.secret/vault-path: "secret/ti-demo-ri"
                 tsi.secret/local-name: "mysecrets/myubuntu-mysecret2"

    In this example, there are two secrets requested:

     * `mysecret1` uses role *demo* and the policy type *ti-demo-all*
     * `mysecret2` uses type *ti-demo-r*

    admission.trusted.identity/inject: "true" activates TSI for this deployment and the modified TSI Admission Controller creates and starts the sidecar.

    second architecture 2

  3. The sidecar collects the measurements called claims that define the identity of this application and sends them securely to the signing service.

  4. The signing service signs these claims using the intermediate CA that was installed during the secure bootstrapping of the cluster and returns the JWT token back to the sidecar.
  5. The sidecar requests the annotated secrets from Vault by passing the JWT token along with the request.
  6. Our Authentication Plugin in Vault intercepts the request call, validates the signature and the expiration date on the JWT token, and if everything is OK, it uses the provided claims against the policies to retrieve the secret. If the measurements match the policy, the secret is released to the application.
  7. The sidecar injects the secret to the running container mounting it at a specified location, e.g. mysecrets/myubuntu-mysecret1 in our example.
  8. Application can easily access the secret locally and use it to obtain sensitive data from the database.

As a result, the secret has been delivered to the container run-time memory, without being ever stored anywhere in Kubernetes, but from the point of view of the application, there were no additional changes needed.

Here is a sample JWT token created by TSI. Notice its three parts: The header, payload containing the actual claims, and a signature for validation.

Sample token

The claims included here represent the measured identity of the application. They contain cluster-region ( e.g. Germany, eu-de), cluster-name, individual machineid which is a unique worker node ID, a unique pod ID, and the list of images making up the pod. These images include the image signature, so you can validate the image and guarantee that the application is running the code that you want to be running and it was not tampered with.

There is also a token expiration timestamp, typically set to one minute, to make these tokens ephemeral and short-lived, protecting the security from leaking.

These are the runtime measurements that represent the identity of the application, signed with the root of trust, and used for evaluation against policies controlling the secrets.

Sample policies and keys

The secrets stored in Vault are protected by policies. Policies are composed of the policy-type and the attributes, the same that are used for building claims, and they represent the path to the secret. If the claims provided in the request matches the policy attribute path, the secret will be released to the application.

Here are sample policies created in Vault:

Sample policies

The top policy, ti-demo-all uses all the main attributes from the claims: cluster-region, cluster-name, namespace and image to build a path for the secret. In this example, the policy allows read access to the secret.

The second policy defines a path using only two measurements, cluster-region and image. Simply put, it allows read access to the secret as long as the container is running from a specific region and it was instantiated from a specific image.

Now, let’s try to create a few sample keys, that are using these policies. First, we want to create a key with the path matching the top policy, using trusted-identity default namespace. The format would be a following:

vault kv put secret/ti-demo-all/${REGION}/${CLUSTER}/trusted-
identity/${IMGSHA}/mysecret1 secret=${SECRET}

Here is the actual command for a key to be used by a container running in Germany (eu-de) in EUCluster using ubuntu 18.04 image with a specific signature — 250cc6f3f3ffc5cdaa9d8f4946ac79821aafb4d3afc93928f0de9336eba21aa4 — for the container.

# For Mac OSX
UIMGSHA=$(echo -n "$UIMG" | shasum -a 256 | awk '{print $1}')
vault kv put secret/ti-demo-all/${REGION}/${CLUSTER}/trusted-identity/${UIMGSHA}/mysecret1 secret=${SECRET}

And here is an example of creating a key for the second policy format:

vault kv put secret/ti-demo-ri/${REGION}/${UIMGSHA}/mysecret2 secret=${SECRET2}

Once the keys are stored in the Vault, the container annotated as in the example above, running from eu-de, in cluster EUCluster with an image ubuntu.18.04 matching the image signature, will get the mysecret1 and mysecret2 injected in specified locations.

Separation of roles

TSI nicely separates functional roles for different personas. The following list details individual functions corresponding to personas used by TSI:

  • Cluster installer — This persona is responsible for bootstrapping TSI before onboarding applications. After the cluster is created, private keys need to be installed on every worker node, then each node needs to be registered with Vault to obtain intermediate CA. This persona requires sysadmin access role to the Kubernetes cluster and Vault admin access to obtain certificates for JWT Signing Service (JSS). Eventually, this role will be replaced by CI/CD bootstrapping pipeline that is deploying the cluster.
  • Application operator — To deploy, operate, and manage the application that is using TSI, this persona requires access to the Kubernetes cluster namespace. The application must be annotated with secret names that will be injected to the container.
  • Policy maker — Typically a Chief Information Security Officer (CISO), this persona manages or approves policies in Vault . He or she defines what policy type and what application measurements are able to obtain the secret. Requires read/write access to Vault.
  • Secret maker — Similar to a policy maker, this persona manages the secret’s life cycle (create, delete, revoke, rotate, etc). Requires read/write access to Vault. This might be replaced by the automation process managing the secrets.

Detailed Overview of TSI

To watch an overview of the TSI, check out this video:

Start using Trusted Service Identity

Now that you know about Trusted Service Identity, start using it.

Check out our GitHub repo and fork the code. All you need is the Kubernetes cluster to use TSI. We have tested TSI on IBM Cloud Kubernetes Service, Minikube, and Red Hat OpenShift on IBM Cloud.

To install, please follow the simple steps outlined here and then follow our Vault sample demo.