Setup guide for Kubernetes developers
So you want to fix Kubernetes?
This setup guide is primarily for developers, technical writers, and testers who are, or will be, working on and contributing to the Kubernetes open source project. For example, it is intended for people who want to make documentation edits, code fixes, and draft new features for Kubernetes; who develop and use various types of applications that use the Kubernetes API, such as profiling, monitoring, and integration tools; and who develop services that specifically take advantage of the Kubernetes feature set.
The beginning section explains the basics. The next sections help you get your development environment set up. Then, you learn about how to use GitHub and Git with the Kubernetes project. The final sections give advice on development, building, and testing, so you can contribute useful code to the Kubernetes project.
Some basic questions up front
Before we get started, you might have a few questions, if you are new to Kubernetes.
What is Kubernetes?
Kubernetes is a set of applications written in go. These applications work together to form an interactive platform for managing containers.
Where is the Kubernetes source code?
Kubernetes is available on GitHub.
Is Kubernetes OSS?
Kubernetes is open source software and was donated by Google to the Cloud Native Computing Foundation (CNCF).
Where does Kubernetes run?
Kubernetes runs on many operating systems/distributions. However, this guide shows you the setup steps for developing and testing Kubernetes on Ubuntu.
To follow along with this setup guide, you need the following prerequisites:
- A GitHub account
- Basic knowledge of Git, bash, and Linux
Estimated time for the steps in this setup guide is totally up to you, where you are already at in theses sequence of steps. Some developers will begin and use the skip-ahead links, perhaps just use this article as a reference. Others might take half a day, a full day, or even a couple days or a week, for example, if you are having a lot of issues or are new to OSS development, testing, or Linux.
To resolve such time variances, I provide the skip-ahead spots. When certain things take a long time, even for experts, I tell you. I also provide shortcuts, which you can take to build and test only very small portions, thus making it fast. How long operations take can depend on your hardware. For example, some of the testing operations might take 10 minutes for a fast cluster or 20 hours on a slow laptop.
Your development environment
Start by creating the basic Kubernetes development environment:
- A Linux terminal, ssh, or both. (See Setting up your host operating system, Setting up Ubuntu, and Using and configuring ssh.)
- Go (golang.org) (See Installing Go.)
- Docker (See Installing Docker and containerd.)
- Build tools (See Installing build tools.)
- jq, a JSON processor (See Installing jq.)
- python-pip and pyyaml (See Installing python-pip and pyyaml.)
If you already have all that set up the way you like it, feel free to skip ahead to GitHub and the Kubernetes project.
Setting up your host operating system
First thing you will need to do is install Ubuntu somewhere. Ubuntu 16.04 or 18.04 LTS Server or Desktop will suffice. You could also use more recent non-LTS releases, it is up to you. I am running 16.04 LTS Desktop in a virtual machine (VM) on my iMac through the latest Virtual Box, 18.04 LTS Desktop on my Mac laptop through Virtual Box, and 18.04 LTS Server on bare metal servers I assembled with mostly used parts.
You can certainly use virtual machines for Kubernetes development, and I do, so don’t be discouraged if you don’t have the resources to build your own server.
To set up Ubuntu on Virtual Box, complete the following steps:
- Download and install Virtual Box: https://www.virtualbox.org/wiki/Downloads
- Download an Ubuntu ISO file (Server/Desktop): http://www.ubuntu.com/
- Create a new vbox using the Ubuntu ISO you just downloaded.
Consider the following suggestions for Virtual Box settings:
- Base memory 8g
- 140g vdi harddrive
- Shared Clipboard — Bidirectional Only (if desktop)
- Video Memory 128mb (if desktop)
- Networking: NAT
- Guest additions must be installed (click VirtualBox VM > Devices > Insert Guest Additions CD Image… if desktop)
Port forwarding rules (see button in advanced Network section):
Setting up Ubuntu
If you are using Ubuntu Desktop, create a terminal:
Then run the software updater:
The following screen capture shows running the
apt-get update from a terminal:
Using and configuring ssh
If you installed Ubuntu desktop and you want to run ssh, complete the following steps to start using ssh:
Use your PC or Mac client to ssh over to your Ubuntu server or desktop. If you installed Ubuntu desktop and you want to run ssh, run the following commands:
$ sudo apt-get install openssh-server $ sudo service ssh start or sudo /etc/init.d/ssh start $ ssh -v localhost
Then log in to your ssh server from your client machine (with a password):
For optional password-less ssh, complete the following steps:
Running the end-to-end Kubernetes tests used to require configuring password-less ssh (using keys not passwords). If you don’t want to set up password-less ssh, skip ahead to Installing Go.
First generate Ubuntu pubic and private keys with the following command:
$ ssh-keygen -t rsa
Then use the default location and enter a password, or just press ENTER:
$ ssh-copy-id firstname.lastname@example.org
On your Mac, run the following command from the
~/.ssh directory to generate a key:
$ ssh-keygen -b 1024 -t rsa -f id_rsa -P ""
Copy and paste your Mac public key (for example,
cat ~/.ssh/d_rsa.pub) into your
Next, configure your Ubuntu guest for password-less ssh:
$ sudo nano /etc/ssh/sshd\_config
Make the following edits to UN-comment/configure:
PubkeyAuthentication yes AuthorizedKeyFile %h/.ssh/authorized\_keys PasswordAuthentication no PermitRootLogin without-password
$ sudo service ssh restart or sudo /etc/init.d/ssh restart
Then, test it:
$ ssh mike@ubuntu (ubuntu = hostname) or 127.0.0.1 or 10.0.2.15 ...
Make sure it works, and then exit.
Because you will log in as root as password-less, you can take a shortcut and copy your user public and private keys, and
authorized_keys over to
/root/.ssh, for example:
$ sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys (you may have to mkdir /root/.ssh) $ sudo cp ~/.ssh/id_rsa /root/.ssh/id_rsa $ sudo cp ~/.ssh/id_rsa.pub /root/.ssh/id_rsa.pub
You should now be able to ssh, root or user, to your guest ubuntu, without a password, from your user account on your Mac terminal:
ssh -p 2222 email@example.com
ssh -p 2222 firstname.lastname@example.org
You should also be able to ssh from your user or root accounts on your Ubuntu host to your user and root accounts on your Ubuntu host:
$ ssh email@example.com $ sudo ssh firstname.lastname@example.org
At the time of installation, make sure to install the latest version of Go that is supported by Kubernetes (or as close as possible).
Pick the version you want from https://golang.org/dl/ and then curl (download) it to your local host:
$ curl -O https://storage.googleapis.com/golang/go1.13.9.linux-amd64.tar.gz
Complete the following installation steps:
$ tar -xvf go1.13.9.linux-amd64.tar.gz $ sudo mv go /usr/local $ rm go1.13.9.linux-amd64.tar.gz
Configure environment variables for applicable users, for example, the currently logged in user (~):
$ sudo nano ~/.profile
$ sudo nano /root/.profile
Now you add the export path statement to those files by adding this line:
$ export PATH=$PATH:/usr/local/go/bin:/home/mike/go/bin
/usr/local/go/bin is for running the go binaries that ship with go, and
/home/username/go/bin is for running any go applications that you built with go.
Test the configuration:
$ go version go version go1.13.9 linux/amd64
To run as sudo with your user profile edits, run the following command:
$ sudo su -- # source /home/mike/.profile
Although riskier, you can log in as root and change your sudo
secure_path through visudo to add paths for go:
Edit the default sudo
Test your sudo configuration:
$ sudo go version go version go1.13.9 linux/amd64
Installing Docker and containerd
For more information, see Install Docker Engine on Ubuntu.
To install Docker, you first update your packages:
$ sudo apt-get update
Then remove any prior versions:
$ sudo apt-get remove docker docker-engine docker.io containerd runc
Install https transport support:
$ sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common
Add Docker’s GPG key:
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Add Docker’s stable releases as an apt repository:
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Install the latest stable versions of Docker and containerd:
$ sudo apt-get update $ sudo apt-get install docker-ce docker-ce-cli containerd.io
To pick a particular version of Docker, run the following command:
$ sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io
Test your Docker installation:
$ sudo docker run hello-world
Add yourself to the Docker group that runs as sudo, so you can run Docker without typing sudo:
$ sudo usermod -aG docker mike
Reboot your Ubuntu host:
$ sudo shutdown -r
Log back into your host with ssh after the host has finished rebooting, and test docker again, without sudo this time:
$ docker run hello-world
Installing build tools
Run the following command:
$ sudo apt-get install build-essential
Run the following command to install jq (a JSON processor):
$ sudo apt-get install jq
Installing python-pip and pyyaml
Run the following commands to install python-pip and pyyaml:
$ sudo apt-get install python-pip $ sudo pip install pyyaml
GitHub and the Kubernetes project
First, let’s look at forking and then cloning your forks. Then there are several considerations for profiles and client configuration. Finally, you learn about the Git workflow for the Kubernetes project.
Create your development forks on GitHub for:
- Kubernetes code at https://github.com/kubernetes/kubernetes
- Kubernetes website and documentation at https://github.com/kubernetes/website
Cloning your forks
Normally, when cloning a github.com source repository, you use the
$GOPATH/src/github.com path or the
~/go/src/github.com/projectname path (representing
the path to source in GitHub). The reason is for convenience when using go
get type commands to pull down GitHub hosted packages.
However, for legacy reasons, Kubernetes packages are named
github.com/kubernetes/kubernetes, as you might expect. Thus, to make all of the development tooling work correctly, to be able to find your package code on disk, you need to put the clone in a
k8s.io directory instead.
k8s.io/kubernetes source trees from your name/kubernetes forks using the following steps:
mike@mike-VirtualBox:~/go/src$ mkdir k8s.io mike@mike-VirtualBox:~/go/src$ cd k8s.io $ git clone https://github.com/mikebrow/kubernetes.git $ git clone https://github.com/mikebrow/website.git $ git clone https://github.com/mikebrow/test-infra.git
Result: The source files are copied (cloned) from your forks into the
k8s.io/website directories. Git creates a
master branch on your local hard disk for your git hub forks.
Check status of your branch:
$ cd kubernetes $ git status On branch master Your branch is up-to-date with 'origin/master`. nothing to commit, working directory clean
GitHub profile recommendations
Consider the following GitHub profile recommendations:
- Add an email to your public profile
- Enable two factor authentication
Git client configuration recommendations
Consider the following client configuration recommendations.
user.* entries to global, local, or both:
$ git config --local user.name "full name here" $ git config --local user.email "email address here"
$ git config --global user.name "full name here" $ git config --global user.email "email address here"
Make it easier to push:
$ git config --global push.default simple
Run the following command so you don’t have to remember your GitHub credentials each time:
$ git config credential.helper store
Tell git to track changes to the cloned branch on your hard disk against kubernetes/kubernetes:
$ git remote add upstream https://github.com/kubernetes/kubernetes.git
git config -l should look like this:
user.name=Mike Brown email@example.com push.default=simple core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=https://github.com/mikebrow/kubernetes.git remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master remote.upstream.url=https://github.com/kubernetes/kubernetes.git remote.upstream.fetch=+refs/heads/*:refs/remotes/upstream/* credential.helper=store
The Git workflow for the Kubernetes open source project
The following diagram shows a typical Git workflow for finding and fixing an issue with an open source product on GitHub. While this flow was drafted for Docker, it also works for Kubernetes.
Find something to work on here in the Kubernetes issues: https://github.com/kubernetes/kubernetes/issues
Make a comment on the issue you found.
If you have questions, don’t be shy.
If you want to work on it, say so. For example, add a sentence that says you are calling
#dibs, to indicate you are working on this issue.
If you are not sure whether you can fix it yourself, at least announce that you are investigating with
#investigating. You are not assigned, but it lets others know that you might call dibs later on.
Kubernetes development (installing etcd)
Before building Kubernetes, you need to install etcd. The following script is included with Kubernetes to install etcd:
Add the displayed path to etcd to your user profiles. For example, add
to your paths with the following command:
$ sudo nano ~/.profile $ sudo nano /root/.profile
And if you want, add it to your
secure\_path for sudo:
$ sudo su - # visudo
Now the secure path should look like this:
Reboot your host:
$ sudo shutdown -r
Log back into your host with ssh after the host has finished rebooting, and test etcd:
$ etcd --version $ sudo etcd --version
Building Kubernetes with make
Use the help target to show make options:
$ make help
A few disclaimers and warnings: The first time you build, verify, and test, Kubernetes takes a long while, because it is building and downloading a significant number of large container images. Some downloads might fail, causing unexpected build and test failures the first and possibly even second time you try to run them. Also since you will be building off “master” errors might be be expected for some or many portions of the testing, and some errors only occur after long timeouts, so patience is required. Even if everything is already downloaded and there are no errors, expect the building, verifying, and testing steps to take a significant amount of time.
If you want to build everything for running Kubernetes locally, use the all target:
$ sudo make all
If you want to build just one of the package/executables, use
make WHAT=cmd/package\_name. For example, to build the kublet server, run the following command:
$ sudo make WHAT=cmd/kublet
Depending on what you changed, the client/server packages might not always build. If you want to force both to be in sync (clean/remove build binaries), use clean:
$ sudo make clean
To generate a release, run the following command:
$ sudo make release
To generate a release without running tests, run the following command:
$ sudo make release-skip-tests
Testing Kubernetes (unit and integration)
Before committing changes to your local development branch, it is recommended that you run verify, unit, and integration tests. Depending on the type of change, you might also want to run end-to-end test buckets.
Build verification tests (for example, presubmit verification) to be done before pushing a PR to
$ sudo make verify
Run the following command for unit tests:
$ sudo make test
Run the following command for just the
$ sudo make test WHAT=./pkg/api/pod
Run just the kubelet tests in verbose mode:
$ sudo make test WHAT=./pkg/kubelet GOFLAGS=-v
Run the pod and kubelet tests in verbose mode:
$ sudo make test WHAT="./pkg/api/pod ./pkg/kubelet" GOFLAGS=-v
Run the following command for integration tests:
$ sudo make test-integration
Run the following command for kubelet integration tests:
$ sudo make test-integration WHAT=./test/integration/kubelet
Run the pod integration tests in verbose mode:
$ sudo make test-integration WHAT=./test/integration/pods GOFLAGS="-v"
Kubernetes end-to-end tests
The following sections show how you run Kubernetes end-to-end, or e2e, tests.
1. Install kubetest
Run the following commands to install kubetest:
$ cd ../test-infra $ sudo GO111MODULE=on go install ./kubetest $ cd ../kubernetes
2. Bring up a local cluster
Running the Kubernetes e2e tests requires using a cluster. The e2e tests can be run on any Kubernetes compatible cluster. This guide shows you how to create a single node cluster running in your Linux environment. If you followed the previous instructions, this cluster runs in your Ubuntu guest account in your VM or Ubuntu host on bare metal. You use two terminals.
A note about end-to-end test: With the default e2e test options, when the version of the client and server are out of sync, the e2e tests won’t run. You can now run the e2e with the versions out of skew by setting the
In terminal 1, build Kubernetes and start up your local cluster:
$ sudo make clean $ sudo make $ sudo PATH=$PATH hack/local-up-cluster.sh
In terminal 2, configure kubectl to interact with your local cluster:
$ export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
Check the status of your single node local cluster:
$ cluster/kubectl.sh get nodes NAME STATUS AGE 127.0.0.1 Ready 3h
3. Run the e2e tests
In terminal 2, run the e2e secrets test bucket:
$ sudo kubetest --provider=local --test --test_args="--minStartupPods=1 --ginkgo.focus=Secrets"
minStartupPods is set to
1 to reflect that you have only one node in your local cluster.
Now run all the e2e tests (this takes a long time to complete):
$ sudo kubetest --provider=local --test --test_args="--minStartupPods=1"
When finished using your Kubernetes cluster, you can quickly bring it down with
ctrl-C in terminal 1.
For more information about end-to-end testing, see e2e-tests.
Kubernetes end-to-end node tests
The following sections show how you run Kubernetes end-to-end note tests, also known as e2e-node tests.
1. Install ginkgo
Run the following commands:
$ sudo go get -u github.com/onsi/ginkgo/ginkgo $ sudo go get -u github.com/onsi/gomega/...
2. Start a cluster
In terminal 1, start up your local single-node cluster:
$ sudo PATH=$PATH hack/local-up-cluster.sh
In terminal 2, configure kubectl to interact with your local cluster:
$ export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
3. Run the e2e-node tests
Sysctls e2e-node tests locally:
$ sudo make test-e2e-node PARALLELISM=1 FOCUS=Sysctls
Run all the
e2e-node tests (This step takes a long time to complete):
$ sudo make test-e2e-node PARALLELISM=1
More information about end-to-end node testing is available at e2e-node-tests document.
Building generated files
After you edit source files that are used in the generation of other source files in the
kubernetes/kubernetes tree, you must update the generated files before doing a git commit. Run the following command:
$ sudo make update Running update-generated-protobuf Running update-codegen Running update-generated-runtime Running update-generated-device-plugin ...
The Kubernetes documentation website
If you want to contribute to the Kubernetes documentation, follow these instructions.
Other things to do
You’re not ready to stop yet. Here are some more tasks you should do:
gofmt -s -w file.goon each changed file before committing your changes.
golinton each changed file before committing your changes. (The Kubernetes team does not currently use lint. But it doesn’t hurt.)
- Remember to update the documentation when creating or modifying features.
- Remember to add test cases to cover your changes.
- Commits that fix or close an issue should reference them in the commit message:
- After every commit, run the test suite and ensure that it passes.
- Sync and rebase frequently as you code, to keep up with the master.
I hope this guide has helped you get started with your journey into Kubernetes development. Review and follow the Kubernetes Community Guidelines to become a successful contributor, member, reviewer, approver (AKA “committer”, AKA “maintainer” ), and/or a sub-project owner.