This blog post will cover setting up a development environment for Kubernetes. This series of posts is primarily for developers, technical writers, and testers who are, or will be, working on the Kubernetes project.
Subsequent posts of this series will cover making specific types of contributions to the Kubernetes project, such as documentation edits, code fixes, and new features; developing and using various types of applications that use the Kubernetes API, such as profiling, monitoring, and integration tools; and developing services that particularly take advantage of the Kubernetes feature set.
What is Kubernetes
Kubernetes is a set of applications written in go. These applications work together to form an interactive platform for managing containers. Kubernetes is also open-source and is available on GitHub. See: https://github.com/kubernetes/kubernetes Kubernetes was donated by Google to the Cloud Native Computing Foundation.
Kubernetes runs on many operating systems. However, for this guide we’ll be showing you how to setup for developing and testing Kubernetes on Ubuntu.
Development Environment Requirements
The basic Kubernetes development environment requires the following:
If you already have all that setup the way you like it, feel free to skip ahead to Forking and Cloning.
Setting up Your Host Operating System on a Virtual Machine
First thing you’ll need to do is install Ubuntu somewhere. Ubuntu 16.04 LTS Server or Desktop will suffice. You could also use 14.04, 15.10 etc., it’s up to you. I’m running Ubuntu 16.04 LTS in a Virtual Machine (VM) on my Mac® via the latest Virtual Box. To set up Ubuntu on Virtual Box:
- Download & Install Virtual Box: https://www.virtualbox.org/wiki/Downloads
- Download an Ubuntu 16.04 ISO file (Server/Desktop): http://www.ubuntu.com/
- Create a new vbox using the Ubuntu ISO you just downloaded.
Virtual Box Settings (Suggestions)
- Base memory 4g
- 80g vdi harddrive
- Shared Clipboard – Bidirectional Only (if desktop)
- Video Memory 128mb (if desktop)
- Networking: NAT
- Guest Additions installed (this is a must)
- VirtualBox VM -> Devices -> Insert Guest Additions CD Image… (if desktop)
- Port Forwarding
Setting up Ubuntu
If you are using Ubuntu Desktop go ahead and create a terminal icon:
Then run software updater:
Running apt-get update from a terminal:
Start Using ssh
Use your pc/mac client to ssh over to your Ubuntu server/desktop. If you installed Ubuntu desktop and you want to run ssh:
$ sudo apt-get install openssh-server
$ sudo service ssh start or sudo /etc/init.d/ssh start
$ ssh -v localhost
Login to your ssh server from your client machine (with a password):
In this example the above specified route handler is forwarding 127.0.0.1:2222 TCP/IP packets to the guest ssh server that is running in the vm. The guest ssh server is listening on port 22 for packets sent to IP address 10.0.2.15. In other setups you might have a different IP address and might not have port forwarding enabled. For example, if the IP address of your guest is 192.168.120.129 (as returned from your terminal via the command
$ hostname -I) and there are no port forwarding rules setup, you can login via:
$ ssh email@example.com
Running the e2e Kubernetes tests requires configuring password-less ssh (using keys not passwords).
First generate Ubuntu pubic and private keys from an Ubuntu terminal:
$ ssh-keygen -t rsa
use the default location and enter a password or just hit enter
$ ssh-copy-id firstname.lastname@example.org
Then on your mac in a mac terminal from the ~/.ssh directory:
$ ssh-keygen -b 1024 -t rsa -f id_rsa -P ""
Copy & paste your mac public key (ex:
cat ~/.ssh/id_rsa.pub) into your ubuntu guest’s
/home/mike/.ssh/authorized_keys file. One way way to do this is by ssh’ing into your host and editing the authorized_keys file via nano. Do this step before setting up pass-wordless below. Not much fun trying to type a public key into a terminal editor that doesn’t have cut and paste support 🙂
Next configure your Ubuntu guest for pass-wordless ssh back on your Ubuntu terminal:
$ sudo nano /etc/ssh/sshd_config
$ sudo service ssh restart or
sudo /etc/init.d/ssh restart
$ ssh mike@ubuntu (ubuntu = hostname) or
ssh 127.0.0.1 or
ssh 10.0.2.15 …
Make sure it works then exit
Since we’ll be logging in as root via password-less, let’s take a shortcut and copy your user public and private keys, as well as authorized_keys over to
/root/.ssh, for example:
$ sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys (you may have to
$ sudo cp ~/.ssh/id_rsa /root/.ssh/id_rsa
$ sudo cp ~/.ssh/id_rsa.pub /root/.ssh/id_rsa.pub
You should now be able to ssh, root or user, to your guest ubuntu, without a password, from your user account on your mac terminal:
ssh -p 2222 email@example.com
ssh -p 2222 firstname.lastname@example.org
You should also be able to ssh from your user or root accounts on your ubuntu host to your user and root accounts on your ubuntu host:
$ ssh email@example.com
$ sudo ssh firstname.lastname@example.org
$ sudo curl -O https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
$ sudo tar -xvf go1.6.linux-amd64.tar.gz
$ sudo mv go /usr/local
Configure environment variables:
$ sudo nano ~/.profile
Add these lines:
mike@mike-VirtualBox:~/go$ go version
go version go1.6.1 linux/amd64
To run as root with your user profile edits:
$ sudo su –
# source /home/mike/.profile
Test your configuration:
root@mike-VirtualBox:~# go version
go version go1.6 linux/amd64
To install Docker:
$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates
$ sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
$ sudo nano /etc/apt/sources.list.d/docker.list
File should include the following (for 14.04 lts aka trusty):
deb https://apt.dockerproject.org/repo ubuntu-trusty main
File should include the following (for 16.04 lts aka xenial):
deb https://apt.dockerproject.org/repo ubuntu-xenial main
To install beta versions of Docker use
testing instead of
$ sudo apt-get update
$ sudo apt-get install linux-image-extra-$(uname -r)
$ sudo apt-get install docker-engine
To pick a particular version of docker:
sudo apt-get install docker-engine=1.10.3-0~trusty
$ sudo service docker start
Test your Docker install:
$ sudo docker run hello-world
Add yourself to the docker group which runs as sudo:
$ sudo usermod -aG docker mike
Test your Docker install again:
$ docker run hello-world
$ curl -L https://github.com/coreos/etcd/releases/download/v3.0.4/etcd-v3.0.4-linux-amd64.tar.gz -o etcd-v3.0.4-linux-amd64.tar.gz
$ tar xzvf etcd-v3.0.4-linux-amd64.tar.gz
/home/mike/etcd-v3.0.4-linux-amd64 to your path in your
In a first terminal run:
In a second terminal:
$ etcdctl set mykey "this is a test"
$ etcdctl get mykey
Create your development forks on github for both
Cloning your Forks
Normally, when cloning a github.com source repository, one uses the path $GOPATH/src/(path to source in github). This is for convenience when using go get type commands to pull down github hosted packages. However, for legacy reasons, Kubernetes packages are named k8s.io/kubernetes not github.com/kubernetes/kubernetes as you might expect. Thus, to make all of the development tooling work right and be able to find your package code on disk you’re going to have to put the clone in a k8s.io directory instead.
Create the k8s.io/kubernetes source trees from your name/kubernetes forks using the following steps:
mike@mike-VirtualBox:~/go/src$ mkdir k8s.io
mike@mike-VirtualBox:~/go/src$ cd k8s.io
$ git clone https://github.com/mikebrow/kubernetes.git
$ git clone https://github.com/mikebrow/kubernetes.github.io.git
Result: source files are copied (cloned) from your forks into the
k8s.io/kubernetes.github.io.git directories. Git creates a master branch on your local hard disk for your git hub forks.
Check status of your branch:
$ cd kubernetes
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Github Profile Recommendations
- Add an email to your public profile
- Enable two factor authentication
Git Client Configuration Recommendations
Set user.* entries to global and/or local:
$ git config --local user.name "Full name here"
$ git config --local user.email "email address here"
$ git config --global user.name "Full name here"
$ git config --global user.email "email address here"
Make it easier to push:
$ git config --global push.default simple
So you don’t have to remember your github credentials each time:
$ git config credential.helper store
Add a remote named upstream for fetching / pushing changes between the cloned branch on your hard disk and kubernetes/kubernetes:
$ git remote add upstream https://github.com/kubernetes/kubernetes.git
git config -l should look like this:
The following steps are recommended to be run by the Kubernetes team. Note: The following link adds a very large bucket of build & verify commands to git commits which will be subsequently run when you commit changes against your branch.
$ cd .git/hooks/
$ ln -s ../../hooks/pre-commit .
If you don’t link the pre-commit checks you risk pushing a PR that will not pass muster. For example, if you modified docs on the k8s.io/kubernetes fork, and you did not run the corresponding documentation scripts, you risk checking in improperly modified docs.
The following pic, shows a typical git workflow for finding and fixing an issue with an open source product on GitHub. While this flow was drafted for Docker, it also works for Kubernetes.
Find something to work on here: https://github.com/kubernetes/kubernetes/issues
Make a comment on the issue you found.
- If you have questions don’t be shy.
- If you want to work on it, say so.
- Adding a #dibs comment to the issue tells others that you are going to work on a resolution to the issue. Calling dibs is a colloquialism meaning others should recognize that you asked to work this issue first. It is the equivalent of raising your hand first to a question given to an audience.
- If you’re not sure whether you can fix it yourself… at least announce that you are investigating with a #investigating comment. You won’t be assigned and it won’t indicate that you have a resolution, but it lets others know that you might call dibs or open up a pull request later on.
To build binaries for running locally:
$ sudo make
e2e Test note: With the default e2e test options, when the version of the client and server are out of sync the e2e tests won’t run. New: You can now set check_version_skew=false, to run the e2e with the versions out of skew.
The client/server may not always build depending on what you changed. To force both to be in sync (clean/remove build binaries) with:
$ sudo make clean
To generate a release:
$ make release
To generate a release without running tests:
$ make release-skip-tests
Testing Kubernetes (unit and integration)
$ sudo make test
Run just the pkg/api tests:
mike@mike-VirtualBox:~/go/src/k8s.io/kubernetes$ sudo WHAT=../../k8s.io/kubernetes/pkg/api make test
$ sudo make test-integration
Kubernetes e2e Tests
Running the Kubernetes e2e tests requires using a cluster. The e2e tests can be run on any Kubernetes compatible cluster. This guide shows you how to create a single node cluster running in your Linux environment. If you followed the above instructions, this cluster will be running in your Ubuntu guest in your VM. We will use two terminals.
In terminal 1, build Kubernetes:
$ sudo make clean
$ sudo make
Before running a local cluster make sure you have cfssl installed:
$ go get -u github.com/cloudflare/cfssl/cmd/...
Bring up your local cluster:
$ sudo PATH=$PATH KUBERNETES_PROVIDER=local hack/local-up-cluster.sh
In terminal 2, configure kubectl to interact with your local cluster:
$ export KUBERNETES_PROVIDER=local
$ cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/apiserver.crt
$ cluster/kubectl.sh config set-credentials myself --username=admin --password=admin
$ cluster/kubectl.sh config set-context local --cluster=local --user=myself
$ cluster/kubectl.sh config use-context local
Check status of your single node native local cluster:
$ cluster/kubectl.sh get nodes
NAME STATUS AGE VERSION
127.0.0.1 Ready 10m v1.6.0-alpha.0.2912+de59ede6b2f4c2-dirty
To configure for running e2e tests on your single node local cluster add the following to your
Check your environment:
mike@mike-VirtualBox:~/go/src/k8s.io/kubernetes$ env | grep kube
To run the e2e tests:
$ KUBE_MASTER_IP=localhost:6443 KUBE_MASTER=https://localhost:6443 go run hack/e2e.go -v --test --test_args="--host=https://localhost:6443"
To run just the e2e Conformance tests:
$ KUBE_MASTER_IP=localhost:6443 KUBE_MASTER=https://localhost:6443 go run hack/e2e.go -v --test --test_args="--host=https://localhost:6443 --ginkgo.focus=through.*Conformance"
When finished using your Kubernetes cluster you can quickly bring it down by hitting ctrl-c in terminal 1.
Running Documentation related scripts
After editing documentation related files in the kubernetes/kubernetes fork use before doing a git commit:
$ sudo hack/update-all.sh
note: requires godep to be installed for sudo as well as $user
Alternatively you can call the following or do a commit and find out which of the following you need to run:
$ sudo hack/update-generated-docs.sh
$ sudo hack/update-generated-swagger-docs.sh
$ sudo hack/update-swagger-spec.sh
Many of the kubernetes md files (documentation files) have been moved from the kubernetes/kubernetes repository to the kubernetes/kubernetes.github.io repository.
For example, the documents published at the documentation web site http://kubernetes.io/docs/ are generated from the kubernetes/kubernetes.github.io repository.
If you followed the earlier instructions you should have a directory named
~/go/src/k8s.io/kubernetes.github.io where these files are located.
Follow these instructions to build and host a local version of the kubernetes documentation website:
In this fashion you can edit the docs and get a preview of what the changes will look like up on kubertnetes.io.
Guidelines/Things to do:
gofmt -s -w file.goon each changed file before committing your changes.
golinton each changed file before committing your changes. (Kubernetes team does not currently use lint. But it doesn’t hurt.)
- Remember to update the documentation when creating or modifying features.
- Remember to add test cases to cover your changes.
- Commits that fix or close an issue should reference them in the commit message Closes #XXXX or Fixes #XXXX.
- After every commit, run the test suite and ensure it is passing.
- Sync and rebase frequently as you code to keep up with master.
I hope this guide helps you get started with your development for Kubernetes.
Cheers, Mike Brown
I’d like to give a special thanks to Douglas Davis and Morgan Bauer for their help in putting this guide together.