Dive into operators, Part 1: Pass configuration to Kubernetes operators with kustomize
Create operators with Ansible and configure the apps they manage
Operators are important for creating, configuring, and managing Kubernetes applications. This tutorial demonstrates a fast and simple way to use the Kubernetes native configuration management tool kustomize to configure operators and the applications they manage.
Prerequisites
Before you walk through this tutorial, you need to set up the following environment:
- Set up a cloud and Kubernetes environment like the IBM Cloud Kubernetes Service.
- Install Operator SDK, as described in Operator SDK Installation.
- Install kustomize, as described in kustomize installation.
Estimated time
Completing this tutorial should take approximately 30 minutes.
Understand operators
Operators enable developers to create, configure, and manage both stateless and stateful Kubernetes applications. An operator’s custom controller watches custom resources specifically defined for the applications. Therefore, an operator mainly consists of Kubernetes CustomResourceDefinitions
(CRDs) and Controller
logic.
With operators, managing complex applications and services becomes easy, but writing an operator is not simple.
Fortunately, the Operator SDK
was introduced as part of the Operator Framework, making it much easier. With Operator SDK, you can bootstrap a new project quickly, and it provides rich high level APIs and extensions for writing operational logics. It includes three types of workflows so you can write operators in Go, Ansible, and Helm.
If you are familiar with Ansible, creating an Ansible type operator with Operator SDK is simple and fast. The scaffolding and code generation are taken care by the SDK. The reconciling logic for the application is driven by the Ansible playbooks and roles, written by users. The operator deployment manifests may be modified to suit the specific operator and application. To configure an operator or the application managed by the operator, you can pass the configurations as environment variables in the operator.yaml
file generated by the Operator SDK.
In this tutorial, you run the Operator SDK command-line interface (CLI) to create an Ansible type operator.
Start by running following command:
operator-sdk new hello-world --api-version=ibm.com/v1alpha1 --kind=Hello --type=ansible
Manage your configuration with kustomize
The kustomize configuration management tool is native to Kubernetes. It offers a template-free way to customize application configuration using plain YAML files. You can install kustomize
as a stand-alone binary file or use it with kubectl
as the apply -k
command.
To run kustomize
with an application project, the project should have directory structure like the following example:
.
├── base
│ ├── crd.yaml
│ ├── kustomization.yaml
│ ├── operator.yaml
│ ├── role.yaml
│ ├── role_binding.yaml
│ └── service_account.yaml
└── overlays
├── production
│ ├── config-map.yaml
│ └── kustomization.yaml
└── staging
├── config-map.yaml
└── kustomization.yaml
The base
directory contains one kustomization.yaml
file and other resource files.
Configure the application with the following command:
kustomize build base > base.yaml
Apply the generated YAML file can to a cluster:
kubectl apply -f base.yaml
To manage variants of configuration, use
overlays
to modify, patch, or merge merge the commonbase
. In each overlay directory, there are onekustomization.yaml
file and other resource files. To generate the final deployment YAML file with a specific overlay, run the following command:kustomize build overlays/production > production.yaml
The generated production.yaml
file contains all the resources from base
and any configuration changes in the production
overlay. You can apply it to a cluster.
Pass configuration to an operator with kustomize
In real world, you can deploy an operator to different clustered environments, such as development, staging, and production. Therefore, an operator requires different configurations. For example, you might deploy an operator to a different namespace and grant it different authorization. Also, the application that an operation manages might take different configurations.
The following example shows how to pass the configuration to an operator using the env
session in the operator Deployment
YAML file.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
replicas: 1
selector:
matchLabels:
name: hello-world
template:
metadata:
labels:
name: hello-world
spec:
serviceAccountName: hello-world
containers:
...
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "hello-world"
- name: DEPLOY_ENV
valueFrom:
configMapKeyRef:
name: install-config
key: DEPLOY_ENV
...
The DEPLOY_ENV
is a configuration to be modified among different cluster deployment. You can use kustomize to change the configuration for different deployments. Complete the following three steps.
Create the
kustomization.yaml
in thebase
directory to generate theinstall-config
ConfigMap, which contains just one config as shown in the following example:apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - crd.yaml - service_account.yaml - role.yaml - role_binding.yaml - operator.yaml commonLabels: kustomize.component: hello-world images: - name: hello-world newName: adrian555/hello-world newTag: v0.0.1 - name: hello-op newName: adrian555/hello-op newTag: v0.0.1 configMapGenerator: - name: install-config literals: - DEPLOY_ENV="base"
Add overlays. In each overlay, update the
DEPLOY_ENV
with a different value:apiVersion: v1 kind: ConfigMap metadata: name: install-config data: DEPLOY_ENV: "production"
Create the
kustomization.yaml
file for the overlay:apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - config-map.yaml
And that is all you need to do.
Examples
Example code for this tutorial is at github.com/adrian555/operator-kustomize.
The hello-image
directory has the Dockerfile to build the image for the Hello World
sample application/service, taken from Kubernetes tutorials.
The hello-world
directory has the operator code generated by the Operator SDK. The reconciling logic is the ansible role in hello-world/roles/hello
directory. Note that the DEPLOY_ENV
configuration is eventually passed on to the application through the operator, in hello-world.yaml.j2
template file.
The hello-kustomize
directory contains the base and overlays YAML files for kustomize. The files in the resources
session of the base/kustomization.yaml
file are copied from the hello-world
‘s deploy
directory (because they are required for deploying the operator).
To deploy the operator with staging
configuration, you run following command:
pushd hello-kustomize/overlays/staging
kustomize build | kubectl apply -f -
popd
Note: You replace overlays/staging
with overlays/production
or base
to generate the specific deployment for different environment. Then the operator should be up and running:
kubectl get pods
## NAME READY STATUS RESTARTS AGE
## hello-world-694cc7b887-lnlcl 2/2 Running 0 20m
To install the application managed by this operator, you apply a CustomResource
:
pushd hello-world/deploy/crds
kubectl apply -f ibm_v1alpha1_hello_cr.yaml
popd
Then you wait until the hello-world
service is up and running:
kubectl get svc
## NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
## hello-world LoadBalancer 172.21.204.30 169.62.90.107 80:31308/TCP 13m
After the hello-world
service is running, you can ping the service to view the output:
curl http://169.62.90.107:31308
## Hello staging!
The service returns the specified config for each kustomize base or overlay setting.
Summary
By following this tutorial, you saw how kustomize is a powerful, Kubernetes-native configuration tool. It simplifies configuration tasks and enhances configurable operators. This tutorial only demonstrated a single configuration with container env
, but with kustomize, you can also patch configurations of other forms, such as json patch
, and runtime data with variables.
Operators are an effective and efficient approach for managing applications. Operators are also Kubernetes applications. To manage them as part of the Operator Framework, Operator Lifecycle Manager (OLM) was introduced. OLM takes care of the operators lifecycle, including updates to the operators and their resources. OLM also is part of the Openshift 4.x Container Platform. Part 2 of this series explains how to combine the power of OLM and kustomize to manage applications.