Example Kubernetes application with an external database
Introduction to a credential rotator scenario that can be automated by using an operator
The best way to understand how useful Kubernetes Operators are is with an example in action. This article introduces you to an application-specific operation within a Kubernetes application that uses an external database.
Sample web application
Our sample web application is deployed in a Kubernetes cluster and uses a back-end database running on a public cloud, as illustrated in the following diagram.
Kubernetes web app with external Cloudant database
The key components include:
- A sample Node.js application deployed on a Kubernetes cluster.
- An IBM Cloudant database running as an IBM Cloud service.
- A resource API key (stored in a Kubernetes Secret on the cluster) that each instance of the web app uses to authenticate with the database. Refer to the Locating your service credentials tutorial within the Cloudant product documentation for more details about the connection information to the Cloudant instance.
Our setup works as expected after the initial deployment of the different components without any additional intervention. However, one common requirement in such applications is to periodically (or on demand in the case of a reported breach of security) change the resource key for the back-end database, which is often called key rotation.
The problem in this scenario is how to automate such a rotation of the database credentials.
Note: Why is it called key-rotation since it is not a recommended practice to reuse old keys (that is, you never wrap around the end of a list of keys)? The answer is kind of lost in time. One view is that the concept of regularly updating the key derives from the traditional use of codebooks, pads, and rotors, where you might change the key every day and, in some cases, you did wrap around and use old keys in effect. In modern systems, we should simply call it key-replacement. But since common parlance is rotation, we will continue to use that term in this learning path.
Problem: Rotating the database credentials
There are a number of steps required to rotate the credentials, such as ensuring that the web application instances are restarted to pick up the new key.
Manual credential rotation
The following manual credential rotation steps are depicted in the previous diagram:
- Create a resource key.
- Update the resource key by writing a new key to the Secret.
- Restart your application instances and wait until it’s up.
- Delete the old resource key.
Those four steps are operations that require a human operator to perform them. Plus, the web application instances must be restarted in a specific order to avoid downtime. How many human operations are required?
While you can simplify the restart step by using the deployment resource to handle all of the restarts, that makes it harder to test the operation of each restarted instance. Testing each instance might let you know whether you need to back off the rotation before all of the instances are switched over and result in a loss of service.
This example demonstrates the necessity of automating the moving parts of an application to avoid outages. Let’s look at how we might solve this problem by using a Kubernetes Operator.
Note: For simplicity, we chose to use an example where the back-end system is a database that is deployed on the same cloud as the application. However, the concept is the same for back-end systems deployed elsewhere. For instance, a SaaS CRM system such as Salesforce that runs in a different location.
Automate by using an operator
The following diagram shows how these steps can be incorporated in a Kubernetes Operator.
Automate by using an operator
Let’s discuss what’s happening in the diagram. The operator is deployed to the cluster like any other Kubernetes application. You can use Kubernetes manifest files or tools such as Helm.
During setup and deployment of the operator:
- A custom resource definition (CRD) is created.
- A controller is deployed as a Pod that contains the operator logic.
- A controller is registered with the controller manager for the CRD defined previously. This means any operation on a custom resource (CR) of this type of CRD is handled by the controller.
- Role-based access control (RBAC) is set up for the CR access.
The new controller enables the credential rotation as follows:
- The user initiates the rotation by creating a CR instance of the new kind we defined, called
- Kubernetes writes the CR to the cluster etcd, just as it does for any resource update.
- The new controller watches for changes to
CredentialRotatorresources, and springs into action.
- The controller creates a service resource key for the back-end service in question. This example uses a Cloudant database deployed in the IBM Cloud.
- The controller updates the Secret with the new resource key.
- The controller restarts the web application instances, which pull the resource key from the Secret.
- The controller deletes the previous resource key for Cloudant in the IBM Cloud.
The Kubernetes cluster is managed by a controller manager that runs controllers in a reconciliation loop in the control plane. Each controller is responsible for managing a specific part of the cluster’s behavior. The controller manager runs a control loop that gives each controller an opportunity to run by invoking its
Reconcile() method. When a controller reconciles, its task is to adjust the current state to make it match the desired state. This is essentially the Kubernetes declarative model. For more details, check out the Get started using Kubernetes Operators learning path.
In the previous example, the controller is called when the CR is modified, which is an instance of the CRD. The controller performs the steps for rotating the database credential and restarting the web application instances.
These steps are all abstracted and contained within the controller, which operates as a native controller in the cluster. How many human operations are required? Only one: creating a CR instance. Kubernetes engages the controller to carry out all of the other steps.
Now that we defined the functionality and steps that we want to carry out, the next section explains how to actually build a controller to complete these steps.