Digital Developer Conference: Cloud Security 2021 -- Build the skills to secure your cloud and data Register free

Example Kubernetes application with an external database

The best way to understand how useful Kubernetes Operators are is with an example in action. This article introduces you to an application-specific operation within a Kubernetes application that uses an external database.

Sample web application

Our sample web application is deployed in a Kubernetes cluster and uses a back-end database running on a public cloud, as illustrated in the following diagram.

Kubernetes web app with external Cloudant database

Diagram of a Kubernetes web app with an external IBM Cloudant database

The key components include:

  • A sample Node.js application deployed on a Kubernetes cluster.
  • An IBM Cloudant database running as an IBM Cloud service.
  • A resource API key (stored in a Kubernetes Secret on the cluster) that each instance of the web app uses to authenticate with the database. Refer to the Locating your service credentials tutorial within the Cloudant product documentation for more details about the connection information to the Cloudant instance.

Our setup works as expected after the initial deployment of the different components without any additional intervention. However, one common requirement in such applications is to periodically (or on demand in the case of a reported breach of security) change the resource key for the back-end database, which is often called key rotation.

The problem in this scenario is how to automate such a rotation of the database credentials.

Note: Why is it called key-rotation since it is not a recommended practice to reuse old keys (that is, you never wrap around the end of a list of keys)? The answer is kind of lost in time. One view is that the concept of regularly updating the key derives from the traditional use of codebooks, pads, and rotors, where you might change the key every day and, in some cases, you did wrap around and use old keys in effect. In modern systems, we should simply call it key-replacement. But since common parlance is rotation, we will continue to use that term in this learning path.

Problem: Rotating the database credentials

There are a number of steps required to rotate the credentials, such as ensuring that the web application instances are restarted to pick up the new key.

Manual credential rotation

Diagram of manual credential rotation steps

The following manual credential rotation steps are depicted in the previous diagram:

  1. Create a resource key.
  2. Update the resource key by writing a new key to the Secret.
  3. Restart your application instances and wait until it’s up.
  4. Delete the old resource key.

Those four steps are operations that require a human operator to perform them. Plus, the web application instances must be restarted in a specific order to avoid downtime. How many human operations are required?

While you can simplify the restart step by using the deployment resource to handle all of the restarts, that makes it harder to test the operation of each restarted instance. Testing each instance might let you know whether you need to back off the rotation before all of the instances are switched over and result in a loss of service.

This example demonstrates the necessity of automating the moving parts of an application to avoid outages. Let’s look at how we might solve this problem by using a Kubernetes Operator.

Note: For simplicity, we chose to use an example where the back-end system is a database that is deployed on the same cloud as the application. However, the concept is the same for back-end systems deployed elsewhere. For instance, a SaaS CRM system such as Salesforce that runs in a different location.

Automate by using an operator

The following diagram shows how these steps can be incorporated in a Kubernetes Operator.

Automate by using an operator

Diagram of an automated credential rotation

Let’s discuss what’s happening in the diagram. The operator is deployed to the cluster like any other Kubernetes application. You can use Kubernetes manifest files or tools such as Helm.

During setup and deployment of the operator:

  • A custom resource definition (CRD) is created.
  • A controller is deployed as a Pod that contains the operator logic.
  • A controller is registered with the controller manager for the CRD defined previously. This means any operation on a custom resource (CR) of this type of CRD is handled by the controller.
  • Role-based access control (RBAC) is set up for the CR access.

The new controller enables the credential rotation as follows:

  1. The user initiates the rotation by creating a CR instance of the new kind we defined, called CredentialRotator.
  2. Kubernetes writes the CR to the cluster etcd, just as it does for any resource update.
  3. The new controller watches for changes to CredentialRotator resources, and springs into action.
  4. The controller creates a service resource key for the back-end service in question. This example uses a Cloudant database deployed in the IBM Cloud.
  5. The controller updates the Secret with the new resource key.
  6. The controller restarts the web application instances, which pull the resource key from the Secret.
  7. The controller deletes the previous resource key for Cloudant in the IBM Cloud.

The Kubernetes cluster is managed by a controller manager that runs controllers in a reconciliation loop in the control plane. Each controller is responsible for managing a specific part of the cluster’s behavior. The controller manager runs a control loop that gives each controller an opportunity to run by invoking its Reconcile() method. When a controller reconciles, its task is to adjust the current state to make it match the desired state. This is essentially the Kubernetes declarative model. For more details, check out the Get started using Kubernetes Operators learning path.

In the previous example, the controller is called when the CR is modified, which is an instance of the CRD. The controller performs the steps for rotating the database credential and restarting the web application instances.

These steps are all abstracted and contained within the controller, which operates as a native controller in the cluster. How many human operations are required? Only one: creating a CR instance. Kubernetes engages the controller to carry out all of the other steps.

Next steps

Now that we defined the functionality and steps that we want to carry out, the next section explains how to actually build a controller to complete these steps.