Digital Developer Conference: Cloud Security 2021 -- Build the skills to secure your cloud and data Register free

The Operator Cookbook: How to make an operator

Previous tutorials in this series showed you how to write a Level I operator for back-end services such as Memcached and JanusGraph. This tutorial provides a general approach to implementing a typical operator. Operators are extremely flexible, so they can be written almost any way you like to do almost anything you want. However, their sweet spot is in managing a single Kubernetes resource or set of resources, often a Deployment or StatefulSet, and sometimes augmented with additional resources such as a Service or PersistentVolumeClaim (PVC). Even if you’re implementing more exotic operators, you’ll first need to understand how to implement these more modest, typical examples.

This tutorial shows you the management logic a service vendor needs to write in order to build an operator. It uses the Operator SDK to develop the operator. It shows how to implement the functionality for a Level I operator, as defined by the Operator Capability Levels. It also describes the common Level I behavior in the Memcached Operator and the JanusGraph Operator, and describes a process for developing most operators this way.

Prerequisites

This tutorial assumes you have some knowledge of Kubernetes Operators concepts but little or no experience developing operators. If you need a refresher, read the “Introduction to Kubernetes Operators” article.

Set up your environment as shown in the “Set up your environment” tutorial.

Estimated time

It should take you about 1 hour to complete this tutorial.

Characteristics of an operator

Let’s briefly review some of the basics about operators.

An operator ensures that the Kubernetes resources that are required to run your service are created and configured properly. It also relays status information back to the user to communicate when the resources are running and describe them.

The Memcached example shows you how to create a Deployment resource for the operator. Once you apply your custom resource (CR) using kubectl, the operator creates a deployment of Memcached — which is the operand, or the application the operator is managing. Similarly, the JanusGraph operator creates a StatefulSet (instead of a Deployment) and exposes it as a Service.

Developing an operator

This tutorial explains the basic steps for developing a typical Level I operator. For this example, it creates an operator for an arbitrary example.com/v1alpha1/Example resource in a fictional GitHub repository, github.com/myorg/example, which is the configuration for instances of an Example service. This example uses a Deployment (with pods) as its managed resource, very much like the Memcached example.

Note: The commands in this tutorial are examples that aren’t meant to work, so don’t run them explicitly. Instead, use the commands and code shown here as templates for how you would develop your own operators.

Any Level I operator that’s implemented using the Operator SDK is developed using these steps:

  1. Initialize the project
  2. Customize the API
  3. Customize the controller
  4. Customize the controller’s reconciliation
  5. Test the operator

There may be other steps for more complex operators, but these steps fit the scaffolding that the SDK generates.

1. Initialize the project

To start, use the SDK to create a project for the new resource type, example.com/v1alpha1/Example, in a Git repo, myorg/example. Your project should use better names than “example”; this process is the template to follow. Here are the commands:

$ operator-sdk init --domain=example.com --repo=github.com/myorg/example-operator
$ operator-sdk create api --group=cache --version=v1alpha1 --kind=Example --controller --resource
Writing scaffold for you to edit...
api/v1alpha1/example_types.go
controllers/example_controller.go

The Operator SDK generates files like api/v1alpha1/example_types.go and controllers/example_controller.go.

For details about these commands, see operator sdk init and operator sdk create api in the previous tutorial.

2. Customize the API

The operator’s API defines the set of variables needed to create operands. All operand instances (such as instances of a database service) start out the same except for the variables. The variables specify how the operator should configure an operand, and the variable values are passed in to the operator via its API.

The SDK generates an API template in a types file. In this example, the file is api/v1alpha1/example_types.go and the contents generated by the SDK look like this:

package v1alpha1
import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type ExampleSpec struct {
    Foo string `json:"foo,omitempty"`
}
type ExampleStatus struct {
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
type Example struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   ExampleSpec   `json:"spec,omitempty"`
    Status ExampleStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

type ExampleList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Example `json:"items"`
}

func init() {
    SchemeBuilder.Register(&Example{}, &ExampleList{})
}

The API generated by the SDK is a scaffold that contains just an example Foo field in the spec and no fields in the status. So the first step in customizing the API is to add some fields.

Customize the spec

Add the fields for the operand’s variables, which are the settings that the operator needs to configure in the operand. For this simple example, let’s add a simple Size field.

The new fields go in the Spec structure in the API. The spec with the Size field looks like this:

// ExampleSpec defines the desired state of Example operand
type ExampleSpec struct {
    Size    int32  `json:"size"`
}

For more details, see “What is the Status?” in the “Build and deploy a basic operator” tutorial.

Customize the status

The operator’s API also needs to know whether the operand has already been created. The API’s Status structure contains that information, typically a list of pods the operator is managing.

By convention, an API stores its list of pods as an array of name strings called Nodes. The status with the Nodes field looks like this:

// ExampleStatus defines the observed state of Example database
type ExampleStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    Nodes []string `json:"nodes"`
}

For more details, see “What is the Status?” in the “Build and deploy a basic operator” tutorial.

Review the resource’s structure

The main structure in the API is the Example resource. This resource consists of the API’s spec and status.

As shown earlier, the SDK has already generated the code for the resource’s structure. Here it is again:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// Example is the Schema for the example API
type Example struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   ExampleSpec   `json:"spec,omitempty"`
    Status ExampleStatus `json:"status,omitempty"`
}

You’ve already updated the API’s spec and status structures, and the API’s resource structure is already built to use those.

Review the resource’s list structure

The API also defines the structure for a list of Example resources.

As shown earlier, the SDK has already generated the code for the resource list’s structure. Here it is again:

// +kubebuilder:object:root=true

// ExampleList contains a list of Example
type ExampleList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Example `json:"items"`
}

The list’s structure is based on the resource’s structure, which is based on the spec and status structures that you customized. You have now customized all of the structures in the API.

3. Customize the controller

The operator’s controller performs reconciliation, which compares the desired state that the API reads from each CR to the managed resources for the operand that are currently defined in the cluster. Where those differ, reconciliation makes changes to the managed resources to make them match the ones requested in the CR.

The SDK generates a controller template in a controller file. In this example, the file is controllers/example_controller.go. The generated controller file contains a structure, ExampleReconciler, and two functions, Reconcile and SetupWithManager:

type ExampleReconciler struct
. . .
func (r *ExampleReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
. . .
func (r *ExampleReconciler) SetupWithManager(mgr ctrl.Manager) error

The reconciler

The controller defines a reconciler structure which it uses to perform reconciliation. The one that the SDK generates is sufficient for reconciliation. It looks like this:

// ExampleReconciler reconciles an Example object
type ExampleReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

This structure is passed into the Reconcile function as the parameter named r, and is used extensively in that function.

Set up with manager

The controller also implements a function, SetupWithManager, to configure itself to listen for changes in the Kubernetes resources that the controller uses — namely the CR and the managed resources. By default, the code generated by the SDK listens to the CR, an instance of example.com/v1alpha1/Example:

// SetupWithManager sets up the controller with the Manager.
func (r *ExampleReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Example{}).
        Complete(r)
}

The SDK does not know what the operator’s managed resources will be — the Reconcile function is empty — so this is all it can generate. When you write the Reconcile function, you implement it to create a single managed resource, a Deployment in this example. The controller needs to listen for changes in that managed resource, so you should add it to SetupWithManager:

// SetupWithManager sets up the controller with the Manager.
func (r *ExampleReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Example{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

Now, SetupWithManager configures the controller to listen for changes not only in the CR, but in the managed resource as well.

Now that you’ve reviewed the ExampleReconciler and customized SetupWithManager, you need to address the more challenging task of customizing the controller’s reconciliation logic.

4. Customize the controller’s reconciliation

The main logic for the operator — logic that must be custom implemented for the operand — is the Reconcile function. The controller performs reconciliation using its Reconciler structure, such as the ExampleReconciler shown earlier.

You can implement reconciliation to do pretty much anything you want, but the Reconcile function for a simple operand typically implements these steps:

  1. Get the CR spec — Read the contents of the CR’s spec field into the API.
  2. Create the operand resources — Create the managed resources if they don’t already exist.
  3. Configure the operand resources — Adjust the managed resources with the variable values from the CR’s spec.
  4. Update the CR status — Use the API to update the CR’s status with relevant details about the managed resources, such as the set of pods.

Here are details on how to implement each of these steps.

4.1 Get the CR spec

The first thing the reconciliation logic typically needs to do is access the specification data in the CR. The controller uses the reconciler to access the CR and load its values into the API.

This is done in just two main lines of code: Create an empty instance of the API’s kind, and use the reconciler to load the API with the contents of the resource in the requested namespace and context. That’s done in this code:

    // Fetch the Example instance
    example := &cachev1alpha1.Example{}
    err := r.Get(ctx, req.NamespacedName, example)

The rest of the code is used to handle errors. If the CR can’t be found, that means it has been deleted since the last time reconciliation was performed — so ignore this request and don’t requeue it. If the error is something else, requeue the request and try again. Here is the error handling code:

    if err != nil {
        if errors.IsNotFound(err) {
            // Request object not found, could have been deleted after reconcile request.
            // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
            // Return and don't requeue.
            log.Info("Example resource not found. Ignoring since object must be deleted.")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request.
        log.Error(err, "Failed to get Example")
        return ctrl.Result{}, err
    }

With that, either the CR’s contents have been loaded into the API variable so you should continue with the next step in reconciliation, or the CR’s contents cannot be accessed so you should abort reconciliation.

4.2 Create the operand resources

Once the API variable contains the CR contents, the next thing for reconciliation to do is create the managed resources if they don’t already exist. If the managed resources do already exist, reconciliation needs access to them.

To do that, like with the CR in the previous step, create an empty instance of the resource’s kind and use the reconciler to load the empty instance with the resources of that kind in the requested namespace and context. That’s done in this code:

found := &appsv1.Deployment{}
err = r.Get(ctx, req.NamespacedName, found)

The logic for finding the managed resources is the same no matter what kind of resource the controller is managing — whether it’s Deployment, StatefulSet, Service, PersistentVolumeClaim (PVC), or something else; whatever kind the controller is managing, the code shown above specifies that kind.

If the resources can be retrieved, then they already exist so the controller proceeds to the next step. Otherwise, if retrieval has run successfully but the resources have not been found, then they don’t exist and need to be created. Otherwise, retrieval has not run successfully so you need to requeue the request and retry the next reconciliation.

Here is the code to create a new managed resource, which again is a Deployment in this case:

        // Define a new deployment
        dep := r.deploymentForExample(example)
        log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
        err = r.Create(ctx, dep)

This creation code is nested in this broader error handling code which determines whether the resource doesn’t exist and whether to requeue the request:

    if err != nil && errors.IsNotFound(err) {
        // Define a new deployment
        dep := r.deploymentForExample(example)
        log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
        err = r.Create(ctx, dep)
        if err != nil {
            log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
            return ctrl.Result{}, err
        }
        // Deployment created successfully - return and requeue
        return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
        log.Error(err, "Failed to get Deployment")
        return ctrl.Result{}, err
    }

Notice that when the managed resource is created successfully, reconciliation ends and requeues the request so that reconciliation is then run again. This gives the cluster time to finish creating the resource before reconciliation tries to configure it in the next step.

With that, the code has either accessed the existing resource or, if there isn’t one, created it and requeued. If the resource already exists, proceed with configuring it.

4.3 Configure the operand resources

Now that the reconciler has the managed resources, reconciliation needs to configure them with the settings specified in the API. When the settings in the API don’t match the managed resources, reconciliation adjusts the managed resources to make them match the settings.

The API defined earlier contains a single field, Size. The managed resources for this example are a single Deployment. In this example schema, Size is equivalent to the number of pod replicas in the Deployment managed resource.

This code retrieves the Size setting from the API:

size := example.Spec.Size

Then this code retrieves the replica size setting from the Deployment managed resource that was accessed (i.e. found) in the previous step:

*found.Spec.Replicas

If these values do not match, then you should set the replica size in the managed resource to the size specified in the API, update the resource, and requeue the request. If the resource can’t be updated (for example, if the cluster is still creating the resource), abort and requeue to try again next reconciliation. The complete set of code looks like this:

// Ensure the deployment size is the same as the spec
size := example.Spec.Size
if *found.Spec.Replicas != size {
    found.Spec.Replicas = &size
    err = r.Update(ctx, found)
    if err != nil {
        log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
        return ctrl.Result{}, err
    }
    // Spec updated - return and requeue
    return ctrl.Result{Requeue: true}, nil
}

With that, the code has updated the managed resources with the configuration specified in the API.

4.4 Update the CR status

The last thing reconciliation typically needs to do is to update the status in the CR. This can be done using the reconciler’s Status().Update() function to update the API, thereby updating the CR.

The status reflects the current set of managed objects and their state. As explained earlier, the managed object is often a Deployment or StatefulSet, in which case the main thing to know about it is its set of pods. The set of pods is stored in the CR in its status field as a list of pod names. It only needs to be updated if the names of the current set of pods is different from the list in the CR.

To start, the controller gets the list of pods in the resource’s namespace with the resource’s labels. It then converts that list of pods into a list of names of those pods, which can be stored in the API.

podList := &corev1.PodList{}
listOpts := []client.ListOption{
    client.InNamespace(example.Namespace),
    client.MatchingLabels(labelsForExample(example.Name)),
}
if err = r.List(ctx, podList, listOpts...); err != nil {
    log.Error(err, "Failed to list pods", "Example.Namespace", example.Namespace, "Example.Name", example.Name)
    return ctrl.Result{}, err
}
podNames := getPodNames(podList.Items)

The controller compares this list of names for the current pods with the list of pod names that the API last knew about. The API’s list of pod names was defined earlier as the string array named Nodes:

// ExampleStatus defines the observed state of Example database
type ExampleStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    Nodes []string `json:"nodes"`
}

The code to retrieve the array of pod names from the API’s status field is:

example.Status.Nodes

If the list of names for the current pods is the same as the API’s list of pod names, then the set of pods hasn’t changed since the last reconciliation so there’s nothing to update in the API’s status. However, if the lists are different, then you need to update the API with the new list and use that to update the CR. This is the code:

// Update status.Nodes if needed
if !reflect.DeepEqual(podNames, example.Status.Nodes) {
    example.Status.Nodes = podNames
    err := r.Status().Update(ctx, example)
    if err != nil {
        log.Error(err, "Failed to update Example status")
        return ctrl.Result{}, err
    }
}

Now that the controller has updated the status, reconciliation is complete.

5. Test the operator

If this were a real operator, to test it, you could confirm that you could scale your operand up or down via the CR. This explains how you could do it using our hypothetical Example operator.

You would tell the operator to scale the operand by changing the size value in its CR:

apiVersion: cache.example.com/v1alpha1
kind: Example
metadata:
  name: example-sample
spec:
  size: 3

Change the size from 3 to 1:

apiVersion: cache.example.com/v1alpha1
kind: Example
metadata:
  name: example-sample
spec:
  size: 1

Once you issue a kubectl apply -f command on the CR, you should see two pods terminating. As long as your application continues to work and is able to scale up and down via the CR, then you have a properly working Level I operator.

Conclusion

Let’s recap. To build a typical operator for a typical backend service hosted in Kubernetes, you need to complete five main tasks:

  1. Set up your operator project using the Operator SDK.
  2. Complete the generated API code to add in the fields needed to specify an operand and track its managed resources.
  3. Customize the generated resources in the controller for managing reconciliation and setting up the controller with the manager.
  4. Customize the controller’s reconciliation logic:
    • Load the CR into the API to get its status.
    • Create or retrieve the managed resources for the operand.
    • Configure the managed resources as described by the API’s spec.
    • Update the CR’s status via the API with details about the managed resources.
  5. Optionally, test the operator by changing the settings in the CR.

Congratulations! You now understand the main concepts behind building a Level I operator. You’re now ready to move on to Level II operators, which are covered in the next tutorial.