Dive into operators, Part 4: Design and create operators based on Knative Common Packages

An operator represents human operational knowledge in software to reliably manage an application, according to a CoreOS definition. When you are working with Kubernetes, an operator is a construct or abstraction that can package, deploy, and manage Kubernetes applications, using standard Kubernetes tools, command-line interface, and APIs. In previous parts of this series, you have learned the basics of the architecture of operators, and some examples of using kustomize and working with the Operator Framework. The Design and create operators based on Kubernetes controller-runtime tutorial shows how to use the Operator Framework, combining the power of OLM and kustomize to manage applications. This tutorial shows an approach with Knative Common Packages.

Remember that when operators are installed, you should have the corresponding Kubernetes applications installed. Kubernetes operators provide capabilities like monitoring, upgrading, scaling, and configuring for applications. If a Kubernetes application is composed by a set of resources or custom resources and the definitions, the Kubernetes operator is a wrapper that can automate the lifecycle management of the resources or custom resources and their definitions.

When you are managing your applications in Kubernetes, the greatest benefit is using Kubernetes capabilities such as porting your resources into Kubernetes APIs by defining CustomResourceDefinitions and implementing the custom controller with the reconcile loop for Create, Read, Update, and Delete (CRUD) operations.

This tutorial walks you through the process of designing and creating your own operator, using a Knative Eventing Operator as an example, built on top of Knative Common Packages. You learn the necessary steps to build your operator from scratch.

Prerequisites

Before you walk through this tutorial, install the following tools:

  • Go, an open source programming language that makes it easy to build software (version 1.13 or later)
  • dep, for managing external Go dependencies.
  • ko, for building, publishing, and running your images in development.
  • kubectl, for managing development environments.
  • (optional)IntelliJ IDEA, an integrated development environment that you can use to view and develop your source code.

In addition, you need to set up a Kubernetes cluster as your environment to run and test your operator. You can use a Kubernetes service from any major cloud provider. For example, if you use the IBM Cloud Kubernetes Service, set up your connection based on the guidance on the website. If you choose to use a local Kubernetes cluster on your own machine, depending on your operating system, you can select Minikube or Docker Desktop.

To start your environment you need to set the following environment variables (add them to your .bashrc):

  • GOPATH: If you don’t have one, simply pick a directory and add export `GOPATH=…“
  • $GOPATH/binon PATH: Install this environment variable so tools installed through go getwill work properly.
  • KO_DOCKER_REPO: This Docker repository is where you push developer images.

Notes: If you are using Docker Hub to store your images, your KO_DOCKER_REPO variable should be docker.io/<username>. Currently Docker Hub doesn’t allow creating subdirs under your user name.

Estimated time

Based on your familiarity with Kubernetes, it might take 15 to 30 minutes for you to successfully create and launch your Kubernetes operator.

Steps

If you look up “Kubernetes Operator” in any search engine, you find plenty of materials that walk you through the process of building an operator, including the previous tutorial in this series. And 99 percent, if not 100 percent of that guidance describes using operator-sdk, which generates all the source code of the operator skeleton based on the controller-runtime package. However, the world will never work solely with a monopoly format. The Kubernetes controller-runtime project is not the only backbone package for building operators. The Knative Common Packages project is an interesting alternative. This tutorial focuses on steps you need to consider for building an operator with the Knative Common Packages.

  1. Create a workspace for your project.

    Because Golang is the primary language to write a Kubernetes operator, you need to create a project for your operator under $GOPATH. You need to have a name for the <project_dir>, which saves all the repositories for a certain project, and a name for the <repo_name>. For example, if you want to create eventing-operator for Knative eventing, the <project_dir> is knative and <repo_name> is eventing-operator.

    Use the following command to create your workspace:

     mkdir -p $GOPATH/src/github.com/<project_dir>/<repo_name>
    

    Replace <project_dir> and <repo_name> with your own project and repo names.

  2. Define a new CustomResourceDefinition (CRD) for the custom resource (CR).

    Because you use an operator to monitor and manage other resources of a Kubernetes application, you need to create a new custom resource, or CR, that manages all of them. Give this CR any name you like. The only condition is it cannot conflict with existing Kubernetes abstractions. To register the CR, define the CRD in a yaml file so it is applied against a Kubernetes cluster.

    After applying this new CRD, the CR becomes a first-class citizen of the Kubernetes cluster, managed the same way you manage other resources. You can define multiple CRs and CRDs based on your need. One special property you need to consider is the scope of the CRD. Should it be scoped by namespace or by cluster? Knative Eventing creates cluster-scoped resources, but the most important deployments are under a specific namespace. I prefer to put the CR of the operator under the same namespace as the deployment, so I define the scope as namespaced. For the eventing-operator, the CRD is put under the config directory. See the example at 300-eventing-v1alpha1-knativeeventing-crd.yaml.

  3. Define the role-based access control (RBAC) of the operator.

    The CR you create in the operator owns all the resources of Knative Eventing and is namespace-scoped. However, the resources of Knative Eventing can be namespace-scoped or cluster-scoped, so you need to define both the Role for namespace-scoped resources, and the ClusterRole for the cluster-scoped resources. You also need to associate verbs with the resources under the apiGroups.

    • To define the Role and ClusterRole, see role.yml under the config directory.

    • Create one service account called knative-eventing-operator for the operator in service_account.yaml. There is only one CR in this operator. It is sufficient to have only one service account.

    • Create the RoleBinding and ClusterRoleBinding to grant the service account the access in role_binding.yaml.

      Place all these files in the config directory.

  4. Create the deployment for the operator.

    You need to have a deployment, which creates a pod running constantly as a service for the eventing operator. Consider some crucial properties like the service account name, the image path for the container, and some environment variables. They are all easy to understand for this operator. See the example manifestfile.

    If you are a fan of the ko command, you do not necessarily need to build and push your image upstream with separate commands. The ko command pushes automatically as long as you specify the path of the main.go. For more information, see github.com/google/ko.

  5. Create the source code under pkg/api/... to extract the CR.

    This file is usually named after <CR name>_types.go. It is an object mapping to the CR defined in YAML format. The main object definition is implemented in the construct of <CR name>Spec, which contains all the attributes in the CR spec. <CR name>Status indicates the status of the CR. Create another file called register.go to register the CR.

    For the Knative eventing-operator, put all the source code files under the pkg/apis/eventing/v1alpha1 directory. Create a file named knativeeventing_types.go, to include the type of the CR, Eventing. You also need to have the EventingSpec type, which defines the desired state, and the EventingStatus type, which indicates the actual state. You also need the EventingList type to map the list of the CRs.

    If you want to generate the openAPI definition file to use in open API spec, add the "+k8s:openapi-gen=true" tag as a comment for the types.

    Because the CR is newly introduced in the Kubernetes, you also need client code to access it through CRUD operations. Add the "+genclient" tag above the type of the CR, which generates default client verb functions (create, update, delete, get, list, update, patch, watch). Depending on the existence of the .Status field in the type, the client is also generated for updateStatus.

    To specify the runtime.Object interface as the interface to generate deepcopy for the type, add the "+k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object" tag.

    To change and retrieve the status of the eventing CR, create a file called knativeeventing_lifecycle.go. Use apis.NewLivingConditionSet in knative/pkg to access and modify the conditions of the CR. These methods are called by the reconcile function.

    To register the CR in Kubernetes, you need to add the newly defined types into the supplied GroupVersionKind scheme. Then you can create a file called register.go.

  6. Generate the source code to register the CR and the client code to access the CR.

    Use two powerful tools: k8s.io/code-generator and knative.dev/pkg(knative common packages) to generate the code of deepcopy,client,informer,lister and injection. All the generations are consolidated in a single script called update-codegen.sh, under the hack directory. You can call this script anytime to generate or update the code under the directory client, where all of generated code is located.

    • Deepcopy means a method func (t* T) DeepCopy() *T for each type T.

    • Client means the typed clientsets for CustomResource APIGroups.

    • Informer offers an event-based interface to react on changes of CustomResources on the server.

    • Lister offers a read-only caching layer for GET and LIST requests.

    • Injection offers a convenient way to access the informer.

      There are three type of clients you can use: a special generated client to access your newly defined CR, a kube client to access the core resources, a kube dynamic client to access the resources through GroupVersionResource or GroupVersionKind. With all the content in update-codegen.sh, you have everything you need to generate.

      Now you set up the dependencies. Create a file named Gopkg.toml under the root directory of your project. See the example Gopkg.toml file for the eventing-operator.

      A script called update-deps.sh under the hack directory generates the dependencies for the project. See the example at update-deps.sh.

      For the first time, run the command dep ensure -v to download the dependencies (because the firectory vendor is empty at the beginning). Then run the ./hack/update-deps.sh command to make sure you only keep the necessary packages as needed.

      You might notice that update-codegen.sh calls the script update-deps.sh. Each time you run update-codegen.sh to generate the source code, you update the dependencies as well.

  7. Specify the resources to be monitored, including the primary CR and any other resource in the Kubernetes application that you care about.

    The CR is the core. Other resources should be registered with this primary CR, so that the custom controller you are creating detects their changes.

    Based on your need, watch the changes to the eventing CR and the changes to the deployments of Knative eventing. With the injection code, you can get the knativeEventingInformer and deploymentInformer, and add them into the event handlers. You can find all the implementation in the controller.go file under pkg/reconciler/knativeeventing/.

    Create a file named reconciler.go in the pkg/reconciler directory. It instantiates an instance with the NewBase function to implement of the common code for the reconciler and adds the CR types into the scheme with init function.

  8. Create custom controller with the implementation of reconcile function.

    The reconcile function implements the crucial business logic. When there is a change detected on the primary CR or any other resources registered, the reconcile function is called. It makes sure that all the resources move to the expected status.

    For a Knative eventing operator, you implement knative eventing installation, knative eventing uninstallation and deployment restoration. Each time the resource is changed, it goes through three steps: initializing the status, installing the knativeeventing by applying the manifest, and verifying the deployment. For details, see knativeeventing.go. Put the in the pkg/reconciler/knativeeventing directory.

  9. Put the manifest of the module to be installed under KO_DATA_PATH.

    If you build your image with the ko command, the default path of KO_DATA_PATH is the cmd/manager/kodata directory. You can put the YAML file of manifest in this directory, if there is one. For the Knative Eventing operator, the example manifest of Knative eventing. If you did not set the environment variable KO_DATA_PATH in the deployment of the operator, the manifest under cmd/manager/kodata is accessed.

  10. To launch the custom controller, call sharedmain.MainWithConfig under the Knative Common Packages.

    The main.go file contains a few lines. With the ko command, you can launch your operator directly from the source code with the ko apply -f config/ command.

Summary

Following the steps in this tutorial, you learned how to design and create your Kubernetes operator, based on Knative Common Packages. You don’t necessarily need to create a Knative project if you want to use the Knative Common Packages project. Because of its friendly license under Apache License 2.0, any project can use Knative Common Packages. Take what you learned here and try it out in your own environment.

Vincent Hou