Digital Developer Conference on Cloud Native Security: Register for free and choose your sessions. June 24, 25, & July 1, 2020 Learn more

Setup guide for Kubernetes developers

This setup guide is primarily for developers, technical writers, and testers who are, or will be, working on and contributing to the Kubernetes open source project. For example, it is intended for people who want to make documentation edits, code fixes, and draft new features for Kubernetes; who develop and use various types of applications that use the Kubernetes API, such as profiling, monitoring, and integration tools; and who develop services that specifically take advantage of the Kubernetes feature set.

The beginning section explains the basics. The next sections help you get your development environment set up. Then, you learn about how to use GitHub and Git with the Kubernetes project. The final sections give advice on development, building, and testing, so you can contribute useful code to the Kubernetes project.

Some basic questions up front

Before we get started, you might have a few questions, if you are new to Kubernetes.

What is Kubernetes?

Kubernetes is a set of applications written in go. These applications work together to form an interactive platform for managing containers.

Where is the Kubernetes source code?

Kubernetes is available on GitHub.

Is Kubernetes OSS?

Kubernetes is open source software and was donated by Google to the Cloud Native Computing Foundation (CNCF).

Where does Kubernetes run?

Kubernetes runs on many operating systems/distributions. However, this guide shows you the setup steps for developing and testing Kubernetes on Ubuntu.

Prerequisites

To follow along with this setup guide, you need the following prerequisites:

  • A GitHub account
  • Basic knowledge of Git, bash, and Linux

Estimated time

Estimated time for the steps in this setup guide is totally up to you, where you are already at in theses sequence of steps. Some developers will begin and use the skip-ahead links, perhaps just use this article as a reference. Others might take half a day, a full day, or even a couple days or a week, for example, if you are having a lot of issues or are new to OSS development, testing, or Linux.

To resolve such time variances, I provide the skip-ahead spots. When certain things take a long time, even for experts, I tell you. I also provide shortcuts, which you can take to build and test only very small portions, thus making it fast. How long operations take can depend on your hardware. For example, some of the testing operations might take 10 minutes for a fast cluster or 20 hours on a slow laptop.

Your development environment

Start by creating the basic Kubernetes development environment:

If you already have all that set up the way you like it, feel free to skip ahead to GitHub and the Kubernetes project.

Setting up your host operating system

First thing you will need to do is install Ubuntu somewhere. Ubuntu 16.04 or 18.04 LTS Server or Desktop will suffice. You could also use more recent non-LTS releases, it is up to you. I am running 16.04 LTS Desktop in a virtual machine (VM) on my iMac through the latest Virtual Box, 18.04 LTS Desktop on my Mac laptop through Virtual Box, and 18.04 LTS Server on bare metal servers I assembled with mostly used parts.

You can certainly use virtual machines for Kubernetes development, and I do, so don’t be discouraged if you don’t have the resources to build your own server.

To set up Ubuntu on Virtual Box, complete the following steps:

Consider the following suggestions for Virtual Box settings:

  • Base memory 8g
  • 140g vdi harddrive
  • Shared Clipboard — Bidirectional Only (if desktop)
  • Video Memory 128mb (if desktop)
  • Networking: NAT
  • Guest additions must be installed (click VirtualBox VM > Devices > Insert Guest Additions CD Image… if desktop)
  • Port forwarding rules (see button in advanced Network section):

    Port forwarding rules in Virtual Box

Setting up Ubuntu

If you are using Ubuntu Desktop, create a terminal:

Ubuntu terminal

Then run the software updater:

Ubuntu software updater

The following screen capture shows running the apt-get update from a terminal:

Running the apt-get command from a terminal

Using and configuring ssh

If you installed Ubuntu desktop and you want to run ssh, complete the following steps to start using ssh:

Use your PC or Mac client to ssh over to your Ubuntu server or desktop. If you installed Ubuntu desktop and you want to run ssh, run the following commands:

$ sudo apt-get install openssh-server
$ sudo service ssh start or sudo /etc/init.d/ssh start
$ ssh -v localhost

Then log in to your ssh server from your client machine (with a password):

Log in to ssh server

For optional password-less ssh, complete the following steps:

Running the end-to-end Kubernetes tests used to require configuring password-less ssh (using keys not passwords). If you don’t want to set up password-less ssh, skip ahead to Installing Go.

First generate Ubuntu pubic and private keys with the following command:

$ ssh-keygen -t rsa

Then use the default location and enter a password, or just press ENTER:

$ ssh-copy-id mike@10.0.2.15

On your Mac, run the following command from the ~/.ssh directory to generate a key:

$ ssh-keygen -b 1024 -t rsa -f id_rsa -P ""

Copy and paste your Mac public key (for example, cat ~/.ssh/d_rsa.pub) into your ubuntu guest /home/user/.ssh/authorized_keys file.

Next, configure your Ubuntu guest for password-less ssh:

$ sudo nano /etc/ssh/sshd\_config

Make the following edits to UN-comment/configure:

PubkeyAuthentication yes

AuthorizedKeyFile %h/.ssh/authorized\_keys

PasswordAuthentication no

PermitRootLogin without-password

Restart:

$ sudo service ssh restart or sudo /etc/init.d/ssh restart

Then, test it:

$ ssh mike@ubuntu (ubuntu = hostname) or 127.0.0.1 or 10.0.2.15 ...

Make sure it works, and then exit.

Because you will log in as root as password-less, you can take a shortcut and copy your user public and private keys, and authorized_keys over to /root/.ssh, for example:

$ sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys (you may have to mkdir /root/.ssh)
$ sudo cp ~/.ssh/id_rsa /root/.ssh/id_rsa
$ sudo cp ~/.ssh/id_rsa.pub /root/.ssh/id_rsa.pub

You should now be able to ssh, root or user, to your guest ubuntu, without a password, from your user account on your Mac terminal:

Terminal 1:

ssh -p 2222 root@127.0.0.1

Terminal 2:

ssh -p 2222 mike@127.0.0.1

You should also be able to ssh from your user or root accounts on your Ubuntu host to your user and root accounts on your Ubuntu host:

$ ssh root@10.0.2.15
$ sudo ssh root@10.0.2.15

Installing Go

At the time of installation, make sure to install the latest version of Go that is supported by Kubernetes (or as close as possible).

Pick the version you want from https://golang.org/dl/ and then curl (download) it to your local host:

  $ curl -O https://storage.googleapis.com/golang/go1.13.9.linux-amd64.tar.gz

Complete the following installation steps:

$ tar -xvf go1.13.9.linux-amd64.tar.gz
$ sudo mv go /usr/local
$ rm go1.13.9.linux-amd64.tar.gz

Configure environment variables for applicable users, for example, the currently logged in user (~):

$ sudo nano ~/.profile

and

$ sudo nano /root/.profile

Now you add the export path statement to those files by adding this line:

$ export PATH=$PATH:/usr/local/go/bin:/home/mike/go/bin

Note: /usr/local/go/bin is for running the go binaries that ship with go, and /home/username/go/bin is for running any go applications that you built with go.

Test the configuration:

$ go version

go version go1.13.9 linux/amd64

To run as sudo with your user profile edits, run the following command:

$ sudo su --

# source /home/mike/.profile

Although riskier, you can log in as root and change your sudo secure_path through visudo to add paths for go:

# visudo

Edit the default sudo secure_path:

Defaults
secure\_path=\"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/go/bin:/home/mike/go/bin\"

Test your sudo configuration:

$ sudo go version

go version go1.13.9 linux/amd64

Installing Docker and containerd

For more information, see Install Docker Engine on Ubuntu.

To install Docker, you first update your packages:

$ sudo apt-get update

Then remove any prior versions:

$ sudo apt-get remove docker docker-engine docker.io containerd runc

Install https transport support:

$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common

Add Docker’s GPG key:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add Docker’s stable releases as an apt repository:

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Install the latest stable versions of Docker and containerd:

$ sudo apt-get update

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

To pick a particular version of Docker, run the following command:

$ sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

Test your Docker installation:

$ sudo docker run hello-world

Add yourself to the Docker group that runs as sudo, so you can run Docker without typing sudo:

$ sudo usermod -aG docker mike

Reboot your Ubuntu host:

$ sudo shutdown -r

Log back into your host with ssh after the host has finished rebooting, and test docker again, without sudo this time:

$ docker run hello-world

Installing build tools

Run the following command:

$ sudo apt-get install build-essential

Installing jq

Run the following command to install jq (a JSON processor):

$ sudo apt-get install jq

Installing python-pip and pyyaml

Run the following commands to install python-pip and pyyaml:

$ sudo apt-get install python-pip

$ sudo pip install pyyaml

GitHub and the Kubernetes project

First, let’s look at forking and then cloning your forks. Then there are several considerations for profiles and client configuration. Finally, you learn about the Git workflow for the Kubernetes project.

Forking

Create your development forks on GitHub for:

Cloning your forks

Normally, when cloning a github.com source repository, you use the $GOPATH/src/github.com path or the ~/go/src/github.com/projectname path (representing the path to source in GitHub). The reason is for convenience when using go get type commands to pull down GitHub hosted packages.

However, for legacy reasons, Kubernetes packages are named k8s.io/kubernetes not github.com/kubernetes/kubernetes, as you might expect. Thus, to make all of the development tooling work correctly, to be able to find your package code on disk, you need to put the clone in a k8s.io directory instead.

Create the k8s.io/kubernetes source trees from your name/kubernetes forks using the following steps:

mike@mike-VirtualBox:~/go/src$ mkdir k8s.io
mike@mike-VirtualBox:~/go/src$ cd k8s.io
$ git clone https://github.com/mikebrow/kubernetes.git
$ git clone https://github.com/mikebrow/website.git
$ git clone https://github.com/mikebrow/test-infra.git

Result: The source files are copied (cloned) from your forks into the go/src/k8s.io/kubernetes and k8s.io/website directories. Git creates a master branch on your local hard disk for your git hub forks.

Check status of your branch:

$ cd kubernetes

$ git status

On branch master
Your branch is up-to-date with 'origin/master`.
nothing to commit, working directory clean

GitHub profile recommendations

Consider the following GitHub profile recommendations:

  • Add an email to your public profile
  • Enable two factor authentication

Git client configuration recommendations

Consider the following client configuration recommendations.

Set the user.* entries to global, local, or both:

$ git config --local user.name "full name here"

$ git config --local user.email "email address here"

or

$ git config --global user.name "full name here"

$ git config --global user.email "email address here"

Make it easier to push:

$ git config --global push.default simple

Run the following command so you don’t have to remember your GitHub credentials each time:

$ git config credential.helper store

Tell git to track changes to the cloned branch on your hard disk against kubernetes/kubernetes:

$ git remote add upstream https://github.com/kubernetes/kubernetes.git

Running git config -l should look like this:

user.name=Mike Brown
user.email=brownwm@us.ibm.com
push.default=simple
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://github.com/mikebrow/kubernetes.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
remote.upstream.url=https://github.com/kubernetes/kubernetes.git
remote.upstream.fetch=+refs/heads/*:refs/remotes/upstream/*
credential.helper=store

The Git workflow for the Kubernetes open source project

The following diagram shows a typical Git workflow for finding and fixing an issue with an open source product on GitHub. While this flow was drafted for Docker, it also works for Kubernetes.

Running the apt-get command from a terminal

Find something to work on here in the Kubernetes issues: https://github.com/kubernetes/kubernetes/issues

Screen capture of issues in the Kubernetes OSS

Make a comment on the issue you found.

If you have questions, don’t be shy.

If you want to work on it, say so. For example, add a sentence that says you are calling #dibs, to indicate you are working on this issue.

If you are not sure whether you can fix it yourself, at least announce that you are investigating with #investigating. You are not assigned, but it lets others know that you might call dibs later on.

Kubernetes development (installing etcd)

Before building Kubernetes, you need to install etcd. The following script is included with Kubernetes to install etcd:

$ hack/install-etcd.sh

Add the displayed path to etcd to your user profiles. For example, add /home/mike/go/src/k8s.io/kubernetes/third_party/etcd to your paths with the following command:

$ sudo nano ~/.profile

$ sudo nano /root/.profile

And if you want, add it to your secure\_path for sudo:

$ sudo su -
# visudo

Now the secure path should look like this:

Defaults
secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/go/bin:/home/mike/go/bin:/home/mike/go/src/k8s.io/kubernetes/third_party/etcd"

Reboot your host:

$ sudo shutdown -r

Log back into your host with ssh after the host has finished rebooting, and test etcd:

$ etcd --version
$ sudo etcd --version

Building Kubernetes with make

Use the help target to show make options:

$ make help

A few disclaimers and warnings: The first time you build, verify, and test, Kubernetes takes a long while, because it is building and downloading a significant number of large container images. Some downloads might fail, causing unexpected build and test failures the first and possibly even second time you try to run them. Also since you will be building off “master” errors might be be expected for some or many portions of the testing, and some errors only occur after long timeouts, so patience is required. Even if everything is already downloaded and there are no errors, expect the building, verifying, and testing steps to take a significant amount of time.

If you want to build everything for running Kubernetes locally, use the all target:

$ sudo make all

If you want to build just one of the package/executables, use make WHAT=cmd/package\_name. For example, to build the kublet server, run the following command:

$ sudo make WHAT=cmd/kublet

Depending on what you changed, the client/server packages might not always build. If you want to force both to be in sync (clean/remove build binaries), use clean:

$ sudo make clean

To generate a release, run the following command:

$ sudo make release

To generate a release without running tests, run the following command:

$ sudo make release-skip-tests

Testing Kubernetes (unit and integration)

Before committing changes to your local development branch, it is recommended that you run verify, unit, and integration tests. Depending on the type of change, you might also want to run end-to-end test buckets.

Build verification tests (for example, presubmit verification) to be done before pushing a PR to kubernetes/kubernetes:

$ sudo make verify

Run the following command for unit tests:

$ sudo make test

Run the following command for just the pkg/api/pod tests:

$ sudo make test WHAT=./pkg/api/pod

Run just the kubelet tests in verbose mode:

$ sudo make test WHAT=./pkg/kubelet GOFLAGS=-v

Run the pod and kubelet tests in verbose mode:

$ sudo make test WHAT="./pkg/api/pod ./pkg/kubelet" GOFLAGS=-v

Run the following command for integration tests:

$ sudo make test-integration

Run the following command for kubelet integration tests:

$ sudo make test-integration WHAT=./test/integration/kubelet

Run the pod integration tests in verbose mode:

$ sudo make test-integration WHAT=./test/integration/pods GOFLAGS="-v"

Kubernetes end-to-end tests

The following sections show how you run Kubernetes end-to-end, or e2e, tests.

1. Install kubetest

Run the following commands to install kubetest:

$ cd ../test-infra
$ sudo GO111MODULE=on go install ./kubetest
$ cd ../kubernetes

2. Bring up a local cluster

Running the Kubernetes e2e tests requires using a cluster. The e2e tests can be run on any Kubernetes compatible cluster. This guide shows you how to create a single node cluster running in your Linux environment. If you followed the previous instructions, this cluster runs in your Ubuntu guest account in your VM or Ubuntu host on bare metal. You use two terminals.

A note about end-to-end test: With the default e2e test options, when the version of the client and server are out of sync, the e2e tests won’t run. You can now run the e2e with the versions out of skew by setting the check_version_skew=false.

In terminal 1, build Kubernetes and start up your local cluster:

$ sudo make clean
$ sudo make
$ sudo PATH=$PATH hack/local-up-cluster.sh

In terminal 2, configure kubectl to interact with your local cluster:

$ export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig

Check the status of your single node local cluster:

$ cluster/kubectl.sh get nodes

NAME        STATUS    AGE
127.0.0.1   Ready     3h

3. Run the e2e tests

In terminal 2, run the e2e secrets test bucket:

$ sudo kubetest --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets"

Note: The minStartupPods is set to 1 to reflect that you have only one node in your local cluster.

Now run all the e2e tests (this takes a long time to complete):

$ sudo kubetest --provider=local --test --test_args="--minStartupPods=1"

When finished using your Kubernetes cluster, you can quickly bring it down with ctrl-C in terminal 1.

For more information about end-to-end testing, see e2e-tests.

Kubernetes end-to-end node tests

The following sections show how you run Kubernetes end-to-end note tests, also known as e2e-node tests.

1. Install ginkgo

Run the following commands:

$ sudo go get -u github.com/onsi/ginkgo/ginkgo
$ sudo go get -u github.com/onsi/gomega/...

2. Start a cluster

In terminal 1, start up your local single-node cluster:

$ sudo PATH=$PATH hack/local-up-cluster.sh

In terminal 2, configure kubectl to interact with your local cluster:

$ export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig

3. Run the e2e-node tests

Run the Sysctls e2e-node tests locally:

$ sudo make test-e2e-node PARALLELISM=1 FOCUS=Sysctls

Run all the e2e-node tests (This step takes a long time to complete):

$ sudo make test-e2e-node PARALLELISM=1

More information about end-to-end node testing is available at e2e-node-tests document.

Building generated files

After you edit source files that are used in the generation of other source files in the kubernetes/kubernetes tree, you must update the generated files before doing a git commit. Run the following command:

$ sudo make update

Running update-generated-protobuf
Running update-codegen
Running update-generated-runtime
Running update-generated-device-plugin
...

The Kubernetes documentation website

If you want to contribute to the Kubernetes documentation, follow these instructions.

Other things to do

You’re not ready to stop yet. Here are some more tasks you should do:

  • Run gofmt -s -w file.go on each changed file before committing your changes.
  • Run golint on each changed file before committing your changes. (The Kubernetes team does not currently use lint. But it doesn’t hurt.)
  • Remember to update the documentation when creating or modifying features.
  • Remember to add test cases to cover your changes.
  • Commits that fix or close an issue should reference them in the commit message: Closes #XXXX or Fixes #XXXX.
  • After every commit, run the test suite and ensure that it passes.
  • Sync and rebase frequently as you code, to keep up with the master.

Summary

I hope this guide has helped you get started with your journey into Kubernetes development. Review and follow the Kubernetes Community Guidelines to become a successful contributor, member, reviewer, approver (AKA “committer”, AKA “maintainer” ), and/or a sub-project owner.

You can reach out to me at brownwm@us.ibm.com and https://twitter.com/mikebrow.

Michael Brown