Develop Knative pipelines on the cloud

The Blog

Blog

Develop Knative pipelines on the cloud

Cloud Containers Continuous Delivery DevOps

by Andrea Frittoli Published March 4, 2019

In this blog post, I will describe how to set up a development environment for Knative’s build pipeline by using your latop and IBM Cloud™, both the container service (IBM Cloud Kubernetes Service) as well as the IBM Container Registry. I won’t get into platform-specific details about how to configure kubectl and other tools on your laptop; instead I will provide links to existing excellent documents. In the next blog post, I will describe how to set up the CD pipeline.

Knative pipelines

Pipelines are the newest addition to the Knative project, which already included three components: serving, eventing, and build. Quoting from the official README, “The Pipeline CRD provides k8s-style resources for declaring CI/CD-style pipelines.” The build-pipeline project introduces a few new custom resource definitions (CRDs) that make it possible to define pipelineresources, tasks/taskruns, and pipelines/pipelineruns directly in Kubernetes.

Preparing your laptop

Before you start, you need to set up the development environment on your laptop. Install git, go and the IBM Cloud CLI. Make sure your GOPATH is set correctly. Either /go or ~/go are good choices, but I prefer the former to keep paths shorter.

# Create the required folders
sudo mkdir -p $GOPATH
sudo chown -R $USER:$USER $GOPATH
mkdir -p ${GOPATH}/src
mkdir -p ${GOPATH}/bin

# Set the following in your shell profile, so that you may download, build and run go programs
export $GOPATH
export PATH=${GOPATH}/bin:$PATH

# Install the IBM Cloud CLI
curl -sL https://ibm.biz/idt-installer | bash

You also need an IBM Cloud account. If you don’t have one, you can create one for free. Knative development benefits from ko to build and deploy its components seamlessly. You will use ko to build knative container images and publish them to the container registry. Let’s go ahead and install it.

go get github.com/google/go-containerregistry/cmd/ko

Next you need to configure ko to push images to the cloud registry:

# Login to the cloud and to the container registry (CR)
ibmcloud login
ibmcloud cr login
REGISTRY_ENDPOINT=$(ibmcloud cr info | awk '/registry/{ print $3 }' | head -1)

# Create a CR token with write access
CR_TOKEN=$(ibmcloud cr token-add --readwrite --description ko_rw --non-expiring -q)
# Backup your docker config if you have one
cp ~/.docker/config.json ~/.docker/config.json.$(date +%F)
# Setup docker auth so it may talk to the CR.
echo '{"auths":{"'$REGISTRY_ENDPOINT'":{"auth":"'$(echo -n token:$CR_TOKEN | base64)'"}}}' > ~/.docker/config.json
# Create a CR namespace
CR_NAMESPACE=knative
ibmcloud cr namespace-add $CR_NAMESPACE
# Configure ko
KO_DOCKER_REPO=$REGISTRY_ENDPOINT/$CR_NAMESPACE

You need a Kubernetes cluster where to deploy Knative. If you don’t have one, provision one in the IBM Cloud Kubernetes Service (IKS). Store the cluster name in the IKS_CLUSTER environment variable.

export IKS_CLUSTER=<your cluster name>
eval $(ibmcloud ks cluster-config $IKS_CLUSTER --export)

Installing Knative from source

Everything is ready now to set up Knative. Obtain the source code:

mkdir -p ${GOPATH}/src/github.com/knative
cd ${GOPATH}/src/github.com/knative
git clone https://github.com/knative/build-pipeline

Deploy Knative build pipeline:

cd ${GOPATH}/src/github.com/knative/build-pipeline
ko apply -f config/

In the last step, ko compiles the code, builds the docker images, pushes them to the registry, updates the YAML manifests to include the correct image path and version and finally applies all of them to the kubernetes cluster. The first time that you run this, it will take a bit longer. The manifest file creates a namespace knative-build-pipeline and a service account within it called build-pipeline-controller. This service account won’t be able to pull the images from the CR until we define the default image pull secret to be used in every pod created with that service account.

# Copy the existing image pull secrets from the default namespace to the knative namespace
kubectl get secret bluemix-default-secret-regional -o yaml | sed 's/default/knative-build-pipeline/g' | kubectl -n knative-build-pipeline create -f -

# Patch the service account to include the secret
kubectl patch -n knative-build-pipeline serviceaccount/build-pipeline-controller -p '{"imagePullSecrets":[{"name": "bluemix-knative-build-pipeline-secret-regional"}]}'

Delete the controller pods so that they are restarted with the right secrets:

kubectl get pods -n knative-build-pipeline | awk '/build/{ print $1 }' | xargs kubectl delete pod -n knative-build-pipeline

If everything went well, you will see something like this:

$ kubectl get all -n knative-build-pipeline
NAME                                             READY   STATUS    RESTARTS   AGE
pod/build-pipeline-controller-85f669c78b-nx7hp   1/1     Running   0          1d
pod/build-pipeline-webhook-7d6dd99bf7-lrzwj      1/1     Running   0          1d

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/build-pipeline-controller   ClusterIP   172.21.137.128           9090/TCP   7d
service/build-pipeline-webhook      ClusterIP   172.21.166.128           443/TCP    7d

NAME                                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/build-pipeline-controller   1         1         1            1           7d
deployment.apps/build-pipeline-webhook      1         1         1            1           7d

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/build-pipeline-controller-7cd4d5495c   0         0         0       5d
replicaset.apps/build-pipeline-controller-85f669c78b   1         1         1       5d
replicaset.apps/build-pipeline-controller-dd945bf4     0         0         0       5d
replicaset.apps/build-pipeline-webhook-684ccc869b      0         0         0       7d
replicaset.apps/build-pipeline-webhook-7d6dd99bf7      1         1         1       5d

Prepare a service account

You configured ko to be able to push images to the registry and the build-pipeline-controller service account to be able to pull images from it. The pipeline will execute the build and push images by using the PIPELINE_SERVICE_ACCOUNT in the PIPELINE_NAMESPACE, so you need to ensure that PIPELINE_SERVICE_ACCOUNT can push images to the registry as well. Create a container registry read/write token, in the same way as you did for configuring ko. Define the following secret template:

apiVersion: v1
kind: Secret
metadata:
  name: ibm-cr-token
  annotations:
    build.knative.dev/docker-0: __CR_ENDPOINT__
type: kubernetes.io/basic-auth
stringData:
  username: token
  password: __CR_TOKEN__

Fill in the endpoint and token values from the environment variables:

# Create the secret manifest
sed -e 's/__CR_TOKEN__/'"$CR_TOKEN"'/g' \
    -e 's/__CR_ENDPOINT__/'"$REGISTRY_ENDPOINT"'/g' \
    cr-secret.yaml.template > cr-secret.yaml

# Create the secret in kubernetes
kubectl apply -f cr-secret.yaml

# Alter the service account to use the secret
PIPELINE_SERVICE_ACCOUNT=default
PIPELINE_NAMESPACE=default
kubectl patch -n $PIPELINE_NAMESPACE serviceaccount/$PIPELINE_SERVICE_ACCOUNT -p '{"secrets":[{"name": "ibm-cr-token"}]}'

Making a code change to Knative

To verify that the development workflow is setup correctly, let’s make a small code change to the Knative pipeline controller:

$ git diff
diff --git a/cmd/controller/main.go b/cmd/controller/main.go
index e6a889ea..34cd26ae 100644
--- a/cmd/controller/main.go
+++ b/cmd/controller/main.go
@@ -63,7 +63,7 @@ func main() {
        logger, atomicLevel := logging.NewLoggerFromConfig(loggingConfig, logging.ControllerLogKey)
        defer logger.Sync()

-       logger.Info("Starting the Pipeline Controller")
+       logger.Info("Starting the Customized Pipeline Controller")

        // set up signals so we handle the first shutdown signal gracefully
        stopCh := signals.SetupSignalHandler()

You can build and deploy the modified controller with just one command:

ko apply -f config/controller.yaml

The output looks like the following:

(...)
2019/02/04 13:03:37 Using base gcr.io/distroless/base:latest for github.com/knative/build-pipeline/cmd/controller
(...)
2019/02/04 13:03:44 Publishing registry.ng.bluemix.net/knative/controller-7cb61323de6451022678822f2a2d2291:latest
2019/02/04 13:03:45 existing blob: sha256:d4210e88ff2398b08758ca768f9230571f8625023c3c59b78b479a26ff2f603d
2019/02/04 13:03:45 existing blob: sha256:bb2297ebc4b391f2fd41c48df5731cdd4dc542f6eb6113436b81c886b139a048
2019/02/04 13:03:45 existing blob: sha256:8ff7789f00584c4605cff901525c8acd878ee103d32351ece7d7c8e5eac5d8b4
2019/02/04 13:03:54 pushed blob: sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0
2019/02/04 13:03:58 pushed blob: sha256:4497e3594708bab98b6f517bd7cfd4a2da18c6c6e3d79731821dd17705bfbee6
2019/02/04 13:03:59 pushed blob: sha256:7aaa1004f57382596bab1f7499bb02e5d1b5b28a288e14e6760ae36b784bf4c0
2019/02/04 13:04:00 registry.ng.bluemix.net/knative/controller-7cb61323de6451022678822f2a2d2291:latest: digest: sha256:f7640cd1e556cc6fe1816d554d7dbd0da1d7d7728f220669e15a00576c999468 size: 918
2019/02/04 13:04:00 Published registry.ng.bluemix.net/knative/controller-7cb61323de6451022678822f2a2d2291@sha256:f7640cd1e556cc6fe1816d554d7dbd0da1d7d7728f220669e15a00576c999468
(...)
deployment.apps/build-pipeline-controller configured

Changing the code of the controller causes the controller pod to be destroyed and recreated automatically, so if you check the controller logs you can see the customized startup message:

$ kubectl logs pod/$(kubectl get pods -n knative-build-pipeline | awk '/controller/{ print $1 }') -n knative-build-pipeline
{"level":"info","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"sampling\": {\n    \"initial\": 100,\n    \"thereafter\": 100\n  },\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","logger":"controller","caller":"controller/main.go:66","msg":"Starting the Customized Pipeline Controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"taskrun/taskrun.go:122","msg":"Setting up event handlers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"taskrun/taskrun.go:134","msg":"Setting up ConfigMap receivers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.pipeline-controller","caller":"pipelinerun/pipelinerun.go:110","msg":"Setting up event handlers","knative.dev/controller":"pipeline-controller"}
W0204 12:04:44.062820       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"level":"info","logger":"controller.taskrun-controller.config-store","caller":"configmap/store.go:166","msg":"taskrun config \"config-entrypoint\" config was added or updated: &{gcr.io/k8s-prow/entrypoint@sha256:7c7cd8906ce4982ffee326218e9fc75da2d4896d53cabc9833b9cc8d2d6b2b8f}","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller","caller":"controller/main.go:143","msg":"Waiting for informer caches to sync"}
{"level":"info","logger":"controller","caller":"controller/main.go:156","msg":"Starting controllers"}
{"level":"info","logger":"controller.pipeline-controller","caller":"controller/controller.go:215","msg":"Starting controller and workers","knative.dev/controller":"pipeline-controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"controller/controller.go:215","msg":"Starting controller and workers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"controller/controller.go:223","msg":"Started workers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.pipeline-controller","caller":"controller/controller.go:223","msg":"Started workers","knative.dev/controller":"pipeline-controller"}

See the fifth line above:

{"level":"info","logger":"controller","caller":"controller/main.go:66","msg":"Starting the Customized Pipeline Controller"}

This means that you successfully set up your Knative pipeline development environment on IBM Cloud! Congratulations!

Notes

The Knative pipeline manifest that configures the build-pipeline-controller service account does not support configuring imagePullSecrets; this is why the service account has to be patched after the initial install. It is convenient, however, when developing on Knative, to simply issue a ko apply -f config/ command to apply to the cluster all code changes at once. That command would however revert the service account and drop the imagePullSecrets. I use git stash to work around this issue as follows:

On a clean code base, alter config/200-serviceaccount.yaml, to include the imagePullSecrets:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-pipeline-controller
  namespace: knative-build-pipeline
imagePullSecrets:
- name: bluemix-knative-build-pipeline-secret-regional

Run git stash to restore a clean code base.

The deployment workflow then becomes:

  # Commit your changes locally, and the pop the service account changes out of stash

  # git add ... / git commit ...
  git stash pop

  # Redeploy
  ko apply -f config

  # Stash the service account change away
  git stash

Conclusions

You can follow a similar approach to set up other Knative components as well. In the next blog post, I will continue by describing how to set up a CD pipeline through the Knative pipeline service that you just installed.

Knative 101: Exercises designed to help you achieve an understanding of Knative

This tutorial introduces you to Knative and where you can access the Knative 101 labs to learn about the platform.

API ManagementCloud+

Blog Feb 08, 2019

Extending Kubernetes for a new developer experience

Kubernetes is only a few years old, but already developers are playing with ways to extend it to suit their…

CloudContainers+

Article Jan 28, 2019

Knative: What is it and why should you care?

These days, many developers are talking about serverless, Kubernetes, and Knative. Let's start by explaining where Knative fits in this…

CloudContainers+

Technologies

Components

Industries

Deployment Models

Solutions

The Blog

Blog