Dive into operators, Part 1: Pass configuration to Kubernetes operators with kustomize

Operators are important for creating, configuring, and managing Kubernetes applications. This tutorial demonstrates a fast and simple way to use the Kubernetes native configuration management tool kustomize to configure operators and the applications they manage.

Prerequisites

Before you walk through this tutorial, you need to set up the following environment:

• Set up a cloud and Kubernetes environment like the IBM Cloud Kubernetes Service.
• Install Operator SDK, as described in Operator SDK Installation.
• Install kustomize, as described in kustomize installation.

Estimated time

Completing this tutorial should take approximately 30 minutes.

Understand operators

Operators enable developers to create, configure, and manage both stateless and stateful Kubernetes applications. An operator’s custom controller watches custom resources specifically defined for the applications. Therefore, an operator mainly consists of Kubernetes CustomResourceDefinitions (CRDs) and Controller logic.

With operators, managing complex applications and services becomes easy, but writing an operator is not simple.

Fortunately, the Operator SDK was introduced as part of the Operator Framework, making it much easier. With Operator SDK, you can bootstrap a new project quickly, and it provides rich high level APIs and extensions for writing operational logics. It includes three types of workflows so you can write operators in Go, Ansible, and Helm.

If you are familiar with Ansible, creating an Ansible type operator with Operator SDK is simple and fast. The scaffolding and code generation are taken care by the SDK. The reconciling logic for the application is driven by the Ansible playbooks and roles, written by users. The operator deployment manifests may be modified to suit the specific operator and application. To configure an operator or the application managed by the operator, you can pass the configurations as environment variables in the operator.yaml file generated by the Operator SDK.

In this tutorial, you run the Operator SDK command-line interface (CLI) to create an Ansible type operator.

Start by running following command:

operator-sdk new hello-world --api-version=ibm.com/v1alpha1 --kind=Hello --type=ansible

Manage your configuration with kustomize

The kustomize configuration management tool is native to Kubernetes. It offers a template-free way to customize application configuration using plain YAML files. You can install kustomize as a stand-alone binary file or use it with kubectl as the apply -k command.

To run kustomize with an application project, the project should have directory structure like the following example:

.
├── base
│   ├── crd.yaml
│   ├── kustomization.yaml
│   ├── operator.yaml
│   ├── role.yaml
│   ├── role_binding.yaml
│   └── service_account.yaml
└── overlays
    ├── production
    │   ├── config-map.yaml
    │   └── kustomization.yaml
    └── staging
        ├── config-map.yaml
        └── kustomization.yaml

The base directory contains one kustomization.yaml file and other resource files.

1. Configure the application with the following command:

    kustomize build base > base.yaml
   

   Show more

2. Apply the generated YAML file can to a cluster:

    kubectl apply -f base.yaml
   

   Show more

3. To manage variants of configuration, use overlays to modify, patch, or merge merge the common base. In each overlay directory, there are one kustomization.yaml file and other resource files. To generate the final deployment YAML file with a specific overlay, run the following command:

    kustomize build overlays/production > production.yaml
   

   Show more

The generated production.yaml file contains all the resources from base and any configuration changes in the production overlay. You can apply it to a cluster.

Pass configuration to an operator with kustomize

In real world, you can deploy an operator to different clustered environments, such as development, staging, and production. Therefore, an operator requires different configurations. For example, you might deploy an operator to a different namespace and grant it different authorization. Also, the application that an operation manages might take different configurations.

The following example shows how to pass the configuration to an operator using the env session in the operator Deployment YAML file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      name: hello-world
  template:
    metadata:
      labels:
        name: hello-world
    spec:
      serviceAccountName: hello-world
      containers:
...
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "hello-world"
            - name: DEPLOY_ENV
              valueFrom:
                configMapKeyRef:
                  name: install-config
                  key: DEPLOY_ENV
...

The DEPLOY_ENV is a configuration to be modified among different cluster deployment. You can use kustomize to change the configuration for different deployments. Complete the following three steps.

1. Create the kustomization.yaml in the base directory to generate the install-config ConfigMap, which contains just one config as shown in the following example:

    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
    - crd.yaml
    - service_account.yaml
    - role.yaml
    - role_binding.yaml
    - operator.yaml
    commonLabels:
      kustomize.component: hello-world
    images:
    - name: hello-world
      newName: adrian555/hello-world
      newTag: v0.0.1
    - name: hello-op
      newName: adrian555/hello-op
      newTag: v0.0.1
    configMapGenerator:
    - name: install-config
      literals:
        - DEPLOY_ENV="base"
   

   Show more

2. Add overlays. In each overlay, update the DEPLOY_ENV with a different value:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: install-config
    data:
      DEPLOY_ENV: "production"
   

   Show more

3. Create the kustomization.yaml file for the overlay:

    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    bases:
    - ../../base
    patchesStrategicMerge:
    - config-map.yaml
   

   Show more

And that is all you need to do.

Examples

Example code for this tutorial is at github.com/adrian555/operator-kustomize.

The hello-image directory has the Dockerfile to build the image for the Hello World sample application/service, taken from Kubernetes tutorials.

The hello-world directory has the operator code generated by the Operator SDK. The reconciling logic is the ansible role in hello-world/roles/hello directory. Note that the DEPLOY_ENV configuration is eventually passed on to the application through the operator, in hello-world.yaml.j2 template file.

The hello-kustomize directory contains the base and overlays YAML files for kustomize. The files in the resources session of the base/kustomization.yaml file are copied from the hello-world‘s deploy directory (because they are required for deploying the operator).

To deploy the operator with staging configuration, you run following command:

pushd hello-kustomize/overlays/staging
kustomize build | kubectl apply -f -
popd

Note: You replace overlays/staging with overlays/production or base to generate the specific deployment for different environment. Then the operator should be up and running:

kubectl get pods
## NAME                               READY   STATUS    RESTARTS   AGE
## hello-world-694cc7b887-lnlcl       2/2     Running   0          20m

To install the application managed by this operator, you apply a CustomResource:

pushd hello-world/deploy/crds
kubectl apply -f ibm_v1alpha1_hello_cr.yaml
popd

Then you wait until the hello-world service is up and running:

kubectl get svc
## NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
## hello-world           LoadBalancer   172.21.204.30   169.62.90.107   80:31308/TCP   13m

After the hello-world service is running, you can ping the service to view the output:

curl http://169.62.90.107:31308
## Hello staging!

The service returns the specified config for each kustomize base or overlay setting.

Summary

By following this tutorial, you saw how kustomize is a powerful, Kubernetes-native configuration tool. It simplifies configuration tasks and enhances configurable operators. This tutorial only demonstrated a single configuration with container env, but with kustomize, you can also patch configurations of other forms, such as json patch, and runtime data with variables.

Operators are an effective and efficient approach for managing applications. Operators are also Kubernetes applications. To manage them as part of the Operator Framework, Operator Lifecycle Manager (OLM) was introduced. OLM takes care of the operators lifecycle, including updates to the operators and their resources. OLM also is part of the Openshift 4.x Container Platform. Part 2 of this series explains how to combine the power of OLM and kustomize to manage applications.