Blog Post
Develop Knative pipelines on the cloud
Set up Knative pipelines on IBM Cloud with Kubernetes and IBM Container Registry.
In this blog post, I will describe how to set up a development environment for Knative’s build pipeline by using your latop and IBM Cloud™, both the container service (IBM Cloud Kubernetes Service) as well as the IBM Container Registry. I won’t get into platform-specific details about how to configure kubectl and other tools on your laptop; instead I will provide links to existing excellent documents. In the next blog post, I will describe how to set up the CD pipeline. This post was originally published at andreafrittoli.me on Feb 5th, 2019.
Knative pipelines
Pipelines are the newest addition to the Knative project, which already included three components: serving, eventing, and build. Quoting from the official README, “The Pipeline CRD provides k8s-style resources for declaring CI/CD-style pipelines.” The build-pipeline project introduces a few new custom resource definitions (CRDs) that make it possible to define pipelineresources, tasks/taskruns, and pipelines/pipelineruns directly in Kubernetes.
Preparing your laptop
Before you start, you need to set up the development environment on your laptop. Install git, go and the IBM Cloud CLI. Make sure your GOPATH is set correctly. Either /go or ~/go are good choices, but I prefer the former to keep paths shorter.
# Create the required folders
sudo mkdir -p $GOPATH
sudo chown -R $USER:$USER $GOPATH
mkdir -p ${GOPATH}/src
mkdir -p ${GOPATH}/bin
# Set the following in your shell profile, so that you may download, build and run go programs
export $GOPATH
export PATH=${GOPATH}/bin:$PATH
# Install the IBM Cloud CLI
curl -sL https://raw.githubusercontent.com/IBM-Bluemix/developer-tools-installer/master/linux-installer/idt-installer | bash
You also need an IBM Cloud account. If you don’t have one, you can create one for free. Knative development benefits from ko to build and deploy its components seamlessly. You will use ko to build knative container images and publish them to the container registry. Let’s go ahead and install it.
go get github.com/google/go-containerregistry/cmd/ko
Next you need to configure ko to push images to the cloud registry:
# Login to the cloud and to the container registry (CR)
ibmcloud login
ibmcloud cr login
REGISTRY_ENDPOINT=$(ibmcloud cr info | awk '/registry/{ print $3 }' | head -1)
# Create a CR token with write access
CR_TOKEN=$(ibmcloud cr token-add --readwrite --description ko_rw --non-expiring -q)
# Backup your docker config if you have one
cp ~/.docker/config.json ~/.docker/config.json.$(date +%F)
# Setup docker auth so it may talk to the CR.
echo '{"auths":{"'$REGISTRY_ENDPOINT'":{"auth":"'$(echo -n token:$CR_TOKEN | base64)'"}}}' > ~/.docker/config.json
# Create a CR namespace
CR_NAMESPACE=knative
ibmcloud cr namespace-add $CR_NAMESPACE
# Configure ko
KO_DOCKER_REPO=$REGISTRY_ENDPOINT/$CR_NAMESPACE
You need a Kubernetes cluster where to deploy Knative. If you don’t have one, provision one in the IBM Cloud Kubernetes Service (IKS). Store the cluster name in the IKS_CLUSTER environment variable.
export IKS_CLUSTER=<your cluster name>
eval $(ibmcloud ks cluster-config $IKS_CLUSTER --export)
Installing Knative from source
Everything is ready now to set up Knative. Obtain the source code:
mkdir -p ${GOPATH}/src/github.com/knative
cd ${GOPATH}/src/github.com/knative
git clone https://github.com/tektoncd/pipeline
Deploy Knative build pipeline:
cd ${GOPATH}/src/github.com/knative/build-pipeline
ko apply -f config/
In the last step, ko compiles the code, builds the docker images, pushes them to the registry, updates the YAML manifests to include the correct image path and version and finally applies all of them to the kubernetes cluster. The first time that you run this, it will take a bit longer. The manifest file creates a namespace knative-build-pipeline and a service account within it called build-pipeline-controller. This service account won’t be able to pull the images from the CR until we define the default image pull secret to be used in every pod created with that service account.
# Copy the existing image pull secrets from the default namespace to the knative namespace
kubectl get secret bluemix-default-secret-regional -o yaml | sed 's/default/knative-build-pipeline/g' | kubectl -n knative-build-pipeline create -f -
# Patch the service account to include the secret
kubectl patch -n knative-build-pipeline serviceaccount/build-pipeline-controller -p '{"imagePullSecrets":[{"name": "bluemix-knative-build-pipeline-secret-regional"}]}'
Delete the controller pods so that they are restarted with the right secrets:
kubectl get pods -n knative-build-pipeline | awk '/build/{ print $1 }' | xargs kubectl delete pod -n knative-build-pipeline
If everything went well, you will see something like this:
$ kubectl get all -n knative-build-pipeline
NAME READY STATUS RESTARTS AGE
pod/build-pipeline-controller-85f669c78b-nx7hp 1/1 Running 0 1d
pod/build-pipeline-webhook-7d6dd99bf7-lrzwj 1/1 Running 0 1d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/build-pipeline-controller ClusterIP 172.21.137.128 9090/TCP 7d
service/build-pipeline-webhook ClusterIP 172.21.166.128 443/TCP 7d
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/build-pipeline-controller 1 1 1 1 7d
deployment.apps/build-pipeline-webhook 1 1 1 1 7d
NAME DESIRED CURRENT READY AGE
replicaset.apps/build-pipeline-controller-7cd4d5495c 0 0 0 5d
replicaset.apps/build-pipeline-controller-85f669c78b 1 1 1 5d
replicaset.apps/build-pipeline-controller-dd945bf4 0 0 0 5d
replicaset.apps/build-pipeline-webhook-684ccc869b 0 0 0 7d
replicaset.apps/build-pipeline-webhook-7d6dd99bf7 1 1 1 5d
Prepare a service account
You configured ko to be able to push images to the registry and the build-pipeline-controller service account to be able to pull images from it. The pipeline will execute the build and push images by using the PIPELINE_SERVICE_ACCOUNT in the PIPELINE_NAMESPACE, so you need to ensure that PIPELINE_SERVICE_ACCOUNT can push images to the registry as well. Create a container registry read/write token, in the same way as you did for configuring ko. Define the following secret template:
apiVersion: v1
kind: Secret
metadata:
name: ibm-cr-token
annotations:
build.knative.dev/docker-0: __CR_ENDPOINT__
type: kubernetes.io/basic-auth
stringData:
username: token
password: __CR_TOKEN__
Fill in the endpoint and token values from the environment variables:
# Create the secret manifest
sed -e 's/__CR_TOKEN__/'"$CR_TOKEN"'/g' \
-e 's/__CR_ENDPOINT__/'"$REGISTRY_ENDPOINT"'/g' \
cr-secret.yaml.template > cr-secret.yaml
# Create the secret in kubernetes
kubectl apply -f cr-secret.yaml
# Alter the service account to use the secret
PIPELINE_SERVICE_ACCOUNT=default
PIPELINE_NAMESPACE=default
kubectl patch -n $PIPELINE_NAMESPACE serviceaccount/$PIPELINE_SERVICE_ACCOUNT -p '{"secrets":[{"name": "ibm-cr-token"}]}'
Making a code change to Knative
To verify that the development workflow is setup correctly, let’s make a small code change to the Knative pipeline controller:
$ git diff
diff --git a/cmd/controller/main.go b/cmd/controller/main.go
index e6a889ea..34cd26ae 100644
--- a/cmd/controller/main.go
+++ b/cmd/controller/main.go
@@ -63,7 +63,7 @@ func main() {
logger, atomicLevel := logging.NewLoggerFromConfig(loggingConfig, logging.ControllerLogKey)
defer logger.Sync()
- logger.Info("Starting the Pipeline Controller")
+ logger.Info("Starting the Customized Pipeline Controller")
// set up signals so we handle the first shutdown signal gracefully
stopCh := signals.SetupSignalHandler()
You can build and deploy the modified controller with just one command:
ko apply -f config/controller.yaml
The output looks like the following:
(...)
2019/02/04 13:03:37 Using base gcr.io/distroless/base:latest for github.com/knative/build-pipeline/cmd/controller
(...)
2019/02/04 13:03:44 Publishing registry.ng.bluemix.net/knative/controller-7cb61323de6451022678822f2a2d2291:latest
2019/02/04 13:03:45 existing blob: sha256:d4210e88ff2398b08758ca768f9230571f8625023c3c59b78b479a26ff2f603d
2019/02/04 13:03:45 existing blob: sha256:bb2297ebc4b391f2fd41c48df5731cdd4dc542f6eb6113436b81c886b139a048
2019/02/04 13:03:45 existing blob: sha256:8ff7789f00584c4605cff901525c8acd878ee103d32351ece7d7c8e5eac5d8b4
2019/02/04 13:03:54 pushed blob: sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0
2019/02/04 13:03:58 pushed blob: sha256:4497e3594708bab98b6f517bd7cfd4a2da18c6c6e3d79731821dd17705bfbee6
2019/02/04 13:03:59 pushed blob: sha256:7aaa1004f57382596bab1f7499bb02e5d1b5b28a288e14e6760ae36b784bf4c0
2019/02/04 13:04:00 registry.ng.bluemix.net/knative/controller-7cb61323de6451022678822f2a2d2291:latest: digest: sha256:f7640cd1e556cc6fe1816d554d7dbd0da1d7d7728f220669e15a00576c999468 size: 918
2019/02/04 13:04:00 Published registry.ng.bluemix.net/knative/controller-7cb61323de6451022678822f2a2d2291@sha256:f7640cd1e556cc6fe1816d554d7dbd0da1d7d7728f220669e15a00576c999468
(...)
deployment.apps/build-pipeline-controller configured
Changing the code of the controller causes the controller pod to be destroyed and recreated automatically, so if you check the controller logs you can see the customized startup message:
$ kubectl logs pod/$(kubectl get pods -n knative-build-pipeline | awk '/controller/{ print $1 }') -n knative-build-pipeline
{"level":"info","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n \"level\": \"info\",\n \"development\": false,\n \"sampling\": {\n \"initial\": 100,\n \"thereafter\": 100\n },\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"\",\n \"levelKey\": \"level\",\n \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \"levelEncoder\": \"\",\n \"timeEncoder\": \"\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n }\n}\n"}
{"level":"info","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","logger":"controller","caller":"controller/main.go:66","msg":"Starting the Customized Pipeline Controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"taskrun/taskrun.go:122","msg":"Setting up event handlers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"taskrun/taskrun.go:134","msg":"Setting up ConfigMap receivers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.pipeline-controller","caller":"pipelinerun/pipelinerun.go:110","msg":"Setting up event handlers","knative.dev/controller":"pipeline-controller"}
W0204 12:04:44.062820 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
{"level":"info","logger":"controller.taskrun-controller.config-store","caller":"configmap/store.go:166","msg":"taskrun config \"config-entrypoint\" config was added or updated: &{gcr.io/k8s-prow/entrypoint@sha256:7c7cd8906ce4982ffee326218e9fc75da2d4896d53cabc9833b9cc8d2d6b2b8f}","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller","caller":"controller/main.go:143","msg":"Waiting for informer caches to sync"}
{"level":"info","logger":"controller","caller":"controller/main.go:156","msg":"Starting controllers"}
{"level":"info","logger":"controller.pipeline-controller","caller":"controller/controller.go:215","msg":"Starting controller and workers","knative.dev/controller":"pipeline-controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"controller/controller.go:215","msg":"Starting controller and workers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.taskrun-controller","caller":"controller/controller.go:223","msg":"Started workers","knative.dev/controller":"taskrun-controller"}
{"level":"info","logger":"controller.pipeline-controller","caller":"controller/controller.go:223","msg":"Started workers","knative.dev/controller":"pipeline-controller"}
See the fifth line above:
{"level":"info","logger":"controller","caller":"controller/main.go:66","msg":"Starting the Customized Pipeline Controller"}
This means that you successfully set up your Knative pipeline development environment on IBM Cloud! Congratulations!
Notes
The Knative pipeline manifest that configures the build-pipeline-controller service account does not support configuring imagePullSecrets; this is why the service account has to be patched after the initial install. It is convenient, however, when developing on Knative, to simply issue a ko apply -f config/ command to apply to the cluster all code changes at once. That command would however revert the service account and drop the imagePullSecrets. I use git stash to work around this issue as follows:
On a clean code base, alter
config/200-serviceaccount.yaml, to include theimagePullSecrets:apiVersion: v1 kind: ServiceAccount metadata: name: build-pipeline-controller namespace: knative-build-pipeline imagePullSecrets: - name: bluemix-knative-build-pipeline-secret-regionalRun
git stashto restore a clean code base.
The deployment workflow then becomes:
# Commit your changes locally, and the pop the service account changes out of stash
# git add ... / git commit ...
git stash pop
# Redeploy
ko apply -f config
# Stash the service account change away
git stash
Conclusions
You can follow a similar approach to set up other Knative components as well. In the next blog post, I will continue by describing how to set up a CD pipeline through the Knative pipeline service that you just installed.