Dive into operators, Part 2: Pass configuration to Kubernetes operators with kustomize
Create operators with Ansible and configure the apps they manage
Operators are important for creating, configuring, and managing Kubernetes applications. This tutorial demonstrates a fast and simple way to use the Kubernetes native configuration management tool kustomize to configure operators and the applications they manage. Kustomize offers a template-free way to customize application configuration using plain YAML files.
Before you walk through this tutorial, you need to set up the following environment:
- Set up a cloud and Kubernetes environment like the IBM Cloud Kubernetes Service.
- Install Operator SDK, as described in Operator SDK Installation.
- Install kustomize, as described in kustomize installation.
Completing this tutorial should take approximately 30 minutes.
Operators enable developers to create, configure, and manage both stateless and stateful Kubernetes applications. An operator’s custom controller watches custom resources specifically defined for the applications. Therefore, an operator mainly consists of Kubernetes
CustomResourceDefinitions (CRDs) and
With operators, managing complex applications and services becomes easy, but writing an operator is not simple.
Operator SDK was introduced as part of the Operator Framework, making it much easier. With Operator SDK, you can bootstrap a new project quickly, and it provides rich high level APIs and extensions for writing operational logics. It includes three types of workflows so you can write operators in Go, Ansible, and Helm.
If you are familiar with Ansible, creating an Ansible type operator with Operator SDK is simple and fast. The scaffolding and code generation are taken care by the SDK. The reconciling logic for the application is driven by the Ansible playbooks and roles, written by users. The operator deployment manifests may be modified to suit the specific operator and application. To configure an operator or the application managed by the operator, you can pass the configurations as environment variables in the
operator.yaml file generated by the Operator SDK.
In this tutorial, you run the Operator SDK command-line interface (CLI) to create an Ansible type operator.
Start by running following command:
operator-sdk new hello-world --api-version=ibm.com/v1alpha1 --kind=Hello --type=ansible
Manage your configuration with kustomize
The kustomize configuration management tool is native to Kubernetes. It offers a template-free way to customize application configuration using plain YAML files. You can install
kustomize as a stand-alone binary file or use it with
kubectl as the
apply -k command.
kustomize with an application project, the project should have directory structure like the following example:
. ├── base │ ├── crd.yaml │ ├── kustomization.yaml │ ├── operator.yaml │ ├── role.yaml │ ├── role_binding.yaml │ └── service_account.yaml └── overlays ├── production │ ├── config-map.yaml │ └── kustomization.yaml └── staging ├── config-map.yaml └── kustomization.yaml
base directory contains one
kustomization.yaml file and other resource files.
Configure the application with the following command:
kustomize build base > base.yaml
Apply the generated YAML file can to a cluster:
kubectl apply -f base.yaml
To manage variants of configuration, use
overlaysto modify, patch, or merge merge the common
base. In each overlay directory, there are one
kustomization.yamlfile and other resource files. To generate the final deployment YAML file with a specific overlay, run the following command:
kustomize build overlays/production > production.yaml
production.yaml file contains all the resources from
base and any configuration changes in the
production overlay. You can apply it to a cluster.
Pass configuration to an operator with kustomize
In real world, you can deploy an operator to different clustered environments, such as development, staging, and production. Therefore, an operator requires different configurations. For example, you might deploy an operator to a different namespace and grant it different authorization. Also, the application that an operation manages might take different configurations.
The following example shows how to pass the configuration to an operator using the
env session in the operator
Deployment YAML file.
apiVersion: apps/v1 kind: Deployment metadata: name: hello-world spec: replicas: 1 selector: matchLabels: name: hello-world template: metadata: labels: name: hello-world spec: serviceAccountName: hello-world containers: ... env: - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: OPERATOR_NAME value: "hello-world" - name: DEPLOY_ENV valueFrom: configMapKeyRef: name: install-config key: DEPLOY_ENV ...
DEPLOY_ENV is a configuration to be modified among different cluster deployment. You can use kustomize to change the configuration for different deployments. Complete the following three steps.
basedirectory to generate the
install-configConfigMap, which contains just one config as shown in the following example:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - crd.yaml - service_account.yaml - role.yaml - role_binding.yaml - operator.yaml commonLabels: kustomize.component: hello-world images: - name: hello-world newName: adrian555/hello-world newTag: v0.0.1 - name: hello-op newName: adrian555/hello-op newTag: v0.0.1 configMapGenerator: - name: install-config literals: - DEPLOY_ENV="base"
Add overlays. In each overlay, update the
DEPLOY_ENVwith a different value:
apiVersion: v1 kind: ConfigMap metadata: name: install-config data: DEPLOY_ENV: "production"
kustomization.yamlfile for the overlay:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - config-map.yaml
And that is all you need to do.
Example code for this tutorial is at github.com/adrian555/operator-kustomize.
hello-image directory has the Dockerfile to build the image for the
Hello World sample application/service, taken from Kubernetes tutorials.
hello-world directory has the operator code generated by the Operator SDK. The reconciling logic is the ansible role in
hello-world/roles/hello directory. Note that the
DEPLOY_ENV configuration is eventually passed on to the application through the operator, in
hello-world.yaml.j2 template file.
hello-kustomize directory contains the base and overlays YAML files for kustomize. The files in the
resources session of the
base/kustomization.yaml file are copied from the
deploy directory (because they are required for deploying the operator).
To deploy the operator with
staging configuration, you run following command:
pushd hello-kustomize/overlays/staging kustomize build | kubectl apply -f - popd
Note: You replace
base to generate the specific deployment for different environment. Then the operator should be up and running:
kubectl get pods ## NAME READY STATUS RESTARTS AGE ## hello-world-694cc7b887-lnlcl 2/2 Running 0 20m
To install the application managed by this operator, you apply a
pushd hello-world/deploy/crds kubectl apply -f ibm_v1alpha1_hello_cr.yaml popd
Then you wait until the
hello-world service is up and running:
kubectl get svc ## NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ## hello-world LoadBalancer 172.21.204.30 22.214.171.124 80:31308/TCP 13m
hello-world service is running, you can ping the service to view the output:
curl http://126.96.36.199:31308 ## Hello staging!
The service returns the specified config for each kustomize base or overlay setting.
By following this tutorial, you saw how kustomize is a powerful, Kubernetes-native configuration tool. It simplifies configuration tasks and enhances configurable operators. This tutorial only demonstrated a single configuration with
container env, but with kustomize, you can also patch configurations of other forms, such as
json patch, and runtime data with variables.
Operators are an effective and efficient approach for managing applications. Operators are also Kubernetes applications. To manage them as part of the Operator Framework, Operator Lifecycle Manager (OLM) was introduced. OLM takes care of the operators lifecycle, including updates to the operators and their resources. OLM also is part of the Openshift 4.x Container Platform. The next part in this series, Design and create operators based on Kubernetes controller-runtime, puts the Operator Framework into practice, demonstrating how to combine the power of OLM and kustomize to manage applications.