An operator represents human operational knowledge in software to reliably manage an application, according to a CoreOS definition. When you are working with Kubernetes, an operator is a construct or abstraction that can package, deploy, and manage Kubernetes applications, using standard Kubernetes tools, command-line interface, and APIs. In previous parts of this series, you have learned the basics of the architecture of operators, and some examples of using kustomize and working with the Operator Framework. The Design and create operators based on Kubernetes controller-runtime tutorial shows how to use the Operator Framework, combining the power of OLM and kustomize to manage applications. This tutorial shows an approach with Knative Common Packages.
Remember that when operators are installed, you should have the corresponding Kubernetes applications installed. Kubernetes operators provide capabilities like monitoring, upgrading, scaling, and configuring for applications. If a Kubernetes application is composed by a set of resources or custom resources and the definitions, the Kubernetes operator is a wrapper that can automate the lifecycle management of the resources or custom resources and their definitions.
When you are managing your applications in Kubernetes, the greatest benefit is using Kubernetes capabilities such as porting your resources into Kubernetes APIs by defining
CustomResourceDefinitions and implementing the custom controller with the reconcile loop for Create, Read, Update, and Delete (CRUD) operations.
This tutorial walks you through the process of designing and creating your own operator, using a Knative Eventing Operator as an example, built on top of Knative Common Packages. You learn the necessary steps to build your operator from scratch.
Before you walk through this tutorial, install the following tools:
- Go, an open source programming language that makes it easy to build software (version 1.13 or later)
- dep, for managing external Go dependencies.
- ko, for building, publishing, and running your images in development.
- kubectl, for managing development environments.
- (optional)IntelliJ IDEA, an integrated development environment that you can use to view and develop your source code.
In addition, you need to set up a Kubernetes cluster as your environment to run and test your operator. You can use a Kubernetes service from any major cloud provider. For example, if you use the IBM Cloud Kubernetes Service, set up your connection based on the guidance on the website. If you choose to use a local Kubernetes cluster on your own machine, depending on your operating system, you can select Minikube or Docker Desktop.
To start your environment you need to set the following environment variables (add them to your
GOPATH: If you don’t have one, simply pick a directory and add export `GOPATH=…“
$GOPATH/binon PATH: Install this environment variable so tools installed through
go getwillwork properly.
KO_DOCKER_REPO: This Docker repository is where you push developer images.
Notes: If you are using Docker Hub to store your images, your
KO_DOCKER_REPO variable should be
Currently Docker Hub doesn’t allow creating
subdirs under your user name.
Based on your familiarity with Kubernetes, it might take 15 to 30 minutes for you to successfully create and launch your Kubernetes operator.
If you look up “Kubernetes Operator” in any search engine, you find plenty of materials that walk you through the process of building an operator, including the previous tutorial in this series. And 99 percent, if not 100 percent of that guidance describes using
operator-sdk, which generates all the source code of the operator skeleton based on the controller-runtime package. However, the world will never work solely with a monopoly format. The Kubernetes
controller-runtime project is not the only backbone package for building operators. The Knative Common Packages project is an interesting alternative. This tutorial focuses on steps you need to consider for building an operator with the Knative Common Packages.
Create a workspace for your project.
Because Golang is the primary language to write a Kubernetes operator, you need to create a project for your operator under
$GOPATH. You need to have a name for the
<project_dir>, which saves all the repositories for a certain project, and a name for the
<repo_name>. For example, if you want to create
eventing-operatorfor Knative eventing, the
Use the following command to create your workspace:
mkdir -p $GOPATH/src/github.com/<project_dir>/<repo_name>
<repo_name>with your own project and repo names.
Define a new
CustomResourceDefinition(CRD) for the custom resource (CR).
Because you use an operator to monitor and manage other resources of a Kubernetes application, you need to create a new custom resource, or CR, that manages all of them. Give this CR any name you like. The only condition is it cannot conflict with existing Kubernetes abstractions. To register the CR, define the CRD in a yaml file so it is applied against a Kubernetes cluster.
After applying this new CRD, the CR becomes a first-class citizen of the Kubernetes cluster, managed the same way you manage other resources. You can define multiple CRs and CRDs based on your need. One special property you need to consider is the scope of the CRD. Should it be scoped by namespace or by cluster? Knative Eventing creates cluster-scoped resources, but the most important deployments are under a specific namespace. I prefer to put the CR of the operator under the same namespace as the deployment, so I define the scope as
namespaced. For the
eventing-operator, the CRD is put under the
configdirectory. See the example at 300-eventing-v1alpha1-knativeeventing-crd.yaml.
Define the role-based access control (RBAC) of the operator.
The CR you create in the operator owns all the resources of Knative Eventing and is namespace-scoped. However, the resources of Knative Eventing can be namespace-scoped or cluster-scoped, so you need to define both the
Rolefor namespace-scoped resources, and the
ClusterRolefor the cluster-scoped resources. You also need to associate verbs with the resources under the
To define the
Create one service account called
knative-eventing-operatorfor the operator in
service_account.yaml. There is only one CR in this operator. It is sufficient to have only one service account.
ClusterRoleBindingto grant the service account the access in
Place all these files in the
Create the deployment for the operator.
You need to have a deployment, which creates a pod running constantly as a service for the eventing operator. Consider some crucial properties like the service account name, the image path for the container, and some environment variables. They are all easy to understand for this operator. See the example manifestfile.
If you are a fan of the
kocommand, you do not necessarily need to build and push your image upstream with separate commands. The
kocommand pushes automatically as long as you specify the path of the
main.go. For more information, see github.com/google/ko.
Create the source code under
pkg/api/...to extract the CR.
This file is usually named after
<CR name>_types.go. It is an object mapping to the CR defined in YAML format. The main object definition is implemented in the construct of
<CR name>Spec, which contains all the attributes in the CR spec.
<CR name>Statusindicates the status of the CR. Create another file called
register.goto register the CR.
For the Knative
eventing-operator, put all the source code files under the
pkg/apis/eventing/v1alpha1directory. Create a file named
knativeeventing_types.go, to include the type of the CR,
Eventing. You also need to have the
EventingSpectype, which defines the desired state, and the
EventingStatustype, which indicates the actual state. You also need the
EventingListtype to map the list of the CRs.
If you want to generate the
openAPIdefinition file to use in open API spec, add the
"+k8s:openapi-gen=true"tag as a comment for the types.
Because the CR is newly introduced in the Kubernetes, you also need client code to access it through CRUD operations. Add the
"+genclient"tag above the type of the CR, which generates default client verb functions (create, update, delete, get, list, update, patch, watch). Depending on the existence of the
.Statusfield in the type, the client is also generated for
To specify the
runtime.Objectinterface as the interface to generate
deepcopyfor the type, add the
To change and retrieve the status of the eventing CR, create a file called
knative/pkgto access and modify the conditions of the CR. These methods are called by the reconcile function.
To register the CR in Kubernetes, you need to add the newly defined types into the supplied
GroupVersionKindscheme. Then you can create a file called
Generate the source code to register the CR and the client code to access the CR.
Use two powerful tools:
knative.dev/pkg(knative common packages)to generate the code of
injection. All the generations are consolidated in a single script called
update-codegen.sh, under the
hackdirectory. You can call this script anytime to generate or update the code under the directory client, where all of generated code is located.
method func (t* T) DeepCopy() *Tfor each type
Clientmeans the typed
Informeroffers an event-based interface to react on changes of
CustomResourceson the server.
Listeroffers a read-only caching layer for
Injectionoffers a convenient way to access the
There are three type of clients you can use: a special generated client to access your newly defined CR, a kube client to access the core resources, a kube dynamic client to access the resources through
GroupVersionKind. With all the content in
update-codegen.sh, you have everything you need to generate.
Now you set up the dependencies. Create a file named
Gopkg.tomlunder the root directory of your project. See the example Gopkg.toml file for the
A script called
hackdirectory generates the dependencies for the project. See the example at update-deps.sh.
For the first time, run the command
dep ensure -vto download the dependencies (because the
firectoryvendor is empty at the beginning). Then run the
./hack/update-deps.shcommand to make sure you only keep the necessary packages as needed.
You might notice that
update-codegen.shcalls the script
update-deps.sh. Each time you run
update-codegen.shto generate the source code, you update the dependencies as well.
Specify the resources to be monitored, including the primary CR and any other resource in the Kubernetes application that you care about.
The CR is the core. Other resources should be registered with this primary CR, so that the custom controller you are creating detects their changes.
Based on your need, watch the changes to the eventing CR and the changes to the deployments of Knative eventing. With the injection code, you can get the
deploymentInformer, and add them into the event handlers. You can find all the implementation in the controller.go file under
Create a file named
pkg/reconcilerdirectory. It instantiates an instance with the
NewBasefunction to implement of the common code for the reconciler and adds the CR types into the scheme with
Create custom controller with the implementation of reconcile function.
reconcilefunction implements the crucial business logic. When there is a change detected on the primary CR or any other resources registered, the
reconcilefunction is called. It makes sure that all the resources move to the expected status.
For a Knative eventing operator, you implement knative eventing installation, knative eventing uninstallation and deployment restoration. Each time the resource is changed, it goes through three steps: initializing the status, installing the
knativeeventingby applying the manifest, and verifying the deployment. For details, see knativeeventing.go. Put the in the
Put the manifest of the module to be installed under
If you build your image with the
kocommand, the default path of
cmd/manager/kodatadirectory. You can put the YAML file of manifest in this directory, if there is one. For the Knative Eventing operator, the example manifest of Knative eventing. If you did not set the environment variable
KO_DATA_PATHin the deployment of the operator, the manifest under
To launch the custom controller, call
sharedmain.MainWithConfigunder the Knative Common Packages.
main.gofile contains a few lines. With the
kocommand, you can launch your operator directly from the source code with the
ko apply -f config/command.
Following the steps in this tutorial, you learned how to design and create your Kubernetes operator, based on Knative Common Packages. You don’t necessarily need to create a Knative project if you want to use the Knative Common Packages project. Because of its friendly license under Apache License 2.0, any project can use Knative Common Packages. Take what you learned here and try it out in your own environment.