KubernetesThis blog post will cover setting up a development environment for Kubernetes. This series of posts is primarily for developers, technical writers, and testers who are, or will be, working on the Kubernetes project. Subsequent posts of this series will cover making specific types of contributions to the Kubernetes project, such as documentation edits, code fixes, and new features; developing and using various types of applications that use the Kubernetes API, such as profiling, monitoring, and integration tools; and developing services that particularly take advantage of the Kubernetes feature set.

What is Kubernetes

googles-new-logoKubernetes is a set of applications written in go. These applications work together to form an interactive platform for managing containers. Kubernetes is also open-source and is available on GitHub. See: https://github.com/kubernetes/kubernetes cncf Kubernetes was donated by Google to the Cloud Native Computing Foundation.   Ubuntu Kubernetes runs on many operating systems. However, for this guide we’ll be showing you how to setup for developing and testing Kubernetes on Ubuntu.

Development Environment Requirements

The basic Kubernetes development environment requires the following:
  • Linux terminalLinuxdocker-logo
  • sshGitHub
  • golang
  • Docker
  • Git & GitHub Account
  • godep
If you already have all that setup the way you like it, feel free to skip ahead to Forking and Cloning.

Setting up Your Host Operating System on a Virtual Machine

First thing you’ll need to do is install Ubuntu somewhere. Ubuntu 16.04 LTS Server or Desktop will suffice. You could also use 14.04, 15.10 etc., it’s up to you. I’m running Ubuntu 16.04 LTS in a Virtual Machine (VM) on my Mac® via the lateAppleMacst Virtual Box. To set up Ubuntu on Virtual Box:  
Virtual Box Settings (Suggestions)
  • Base memory 4g
  • 80g vdi harddrive
  • Shared Clipboard – Bidirectional Only (if desktop)
  • Video Memory 128mb (if desktop)
  • Networking: NAT
  • Guest Additions installed (this is a must)
    • VirtualBox VM -> Devices -> Insert Guest Additions CD Image… (if desktop)
  • Port Forwarding

Port Forwarding

Setting up Ubuntu

If you are using Ubuntu Desktop go ahead and create a terminal icon:

desktopterm

Then run software updater:

softwareupdater

Running apt-get update from a terminal:

aptgetupdate

Start Using ssh

Use your pc/mac client to ssh over to your Ubuntu server/desktop. If you installed Ubuntu desktop and you want to run ssh:

$ sudo apt-get install openssh-server $ sudo service ssh start or sudo /etc/init.d/ssh start $ ssh -v localhost

Login to your ssh server from your client machine (with a password):

ssh login

In this example the above specified route handler is forwarding 127.0.0.1:2222 TCP/IP packets to the guest ssh server that is running in the vm. The guest ssh server is listening on port 22 for packets sent to IP address 10.0.2.15. In other setups you might have a different IP address and might not have port forwarding enabled. For example, if the IP address of your guest is 192.168.120.129 (as returned from your terminal via the command $ hostname -I) and there are no port forwarding rules setup, you can login via:

$ ssh mike@192.168.120.129

Configure ssh

Running the e2e Kubernetes tests requires configuring password-less ssh (using keys not passwords). First generate Ubuntu pubic and private keys from an Ubuntu terminal:

$ ssh-keygen -t rsa use the default location and enter a password or just hit enter $ ssh-copy-id mike@10.0.2.15

Then on your mac in a mac terminal from the ~/.ssh directory:

$ ssh-keygen -b 1024 -t rsa -f id_rsa -P "" Copy & paste your mac public key (ex: cat ~/.ssh/id_rsa.pub) into your ubuntu guest’s /home/mike/.ssh/authorized_keys file. One way way to do this is by ssh’ing into your host and editing the authorized_keys file via nano. Do this step before setting up pass-wordless below. Not much fun trying to type a public key into a terminal editor that doesn’t have cut and paste support 🙂

Next configure your Ubuntu guest for pass-wordless ssh back on your Ubuntu terminal:

$ sudo nano /etc/ssh/sshd_config

UN-comment/configure:

PubkeyAuthentication yes AuthorizedKeyFile %h/.ssh/authorized_keys PasswordAuthentication no PermitRootLogin without-password

$ sudo service ssh restart or sudo /etc/init.d/ssh restart

Test it:

$ ssh mike@ubuntu (ubuntu = hostname) or ssh 127.0.0.1 or ssh 10.0.2.15

Make sure it works then exit

Since we’ll be logging in as root via password-less, let’s take a shortcut and copy your user public and private keys, as well as authorized_keys over to /root/.ssh, for example:

$ sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys (you may have to mkdir /root/.ssh) $ sudo cp ~/.ssh/id_rsa /root/.ssh/id_rsa $ sudo cp ~/.ssh/id_rsa.pub /root/.ssh/id_rsa.pub

You should now be able to ssh, root or user, to your guest ubuntu, without a password, from your user account on your mac terminal:

Terminal 1: ssh -p 2222 root@127.0.0.1 Terminal 2: ssh -p 2222 mike@127.0.0.1

You should also be able to ssh from your user or root accounts on your ubuntu host to your user and root accounts on your ubuntu host:

$ ssh root@10.0.2.15 $ sudo ssh root@10.0.2.15

Install go

Download: https://golang.org/dl/

$ sudo curl -O https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz

Extract:

$ sudo tar -xvf go1.6.linux-amd64.tar.gz $ sudo mv go /usr/local

Configure environment variables:

$ sudo nano ~/.profile Add these lines: export GOPATH=/home/mike/go export PATH=$PATH:/usr/local/go/bin:/home/mike/go/bin

Test configuration:

mike@mike-VirtualBox:~/go$ go version go version go1.6.1 linux/amd64

To run as root with your user profile edits:

$ sudo su – # source /home/mike/.profile

Test your configuration:

root@mike-VirtualBox:~# go version go version go1.6 linux/amd64

Install Docker

See: https://docs.docker.com/engine/installation/linux/ubuntulinux/

To install Docker: $ sudo apt-get update $ sudo apt-get install apt-transport-https ca-certificates $ sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D $ sudo nano /etc/apt/sources.list.d/docker.list

File should include the following (for 14.04 lts aka trusty): deb https://apt.dockerproject.org/repo ubuntu-trusty main File should include the following (for 16.04 lts aka xenial): deb https://apt.dockerproject.org/repo ubuntu-xenial main To install beta versions of Docker use testing instead of main.

$ sudo apt-get update $ sudo apt-get install linux-image-extra-$(uname -r) $ sudo apt-get install docker-engine To pick a particular version of docker: sudo apt-get install docker-engine=1.10.3-0~trusty $ sudo service docker start

Test your Docker install:

$ sudo docker run hello-world

Add yourself to the docker group which runs as sudo:

$ sudo usermod -aG docker mike

Relog Test your Docker install again:

$ docker run hello-world

Install etcd

$ curl -L https://github.com/coreos/etcd/releases/download/v3.0.4/etcd-v3.0.4-linux-amd64.tar.gz -o etcd-v3.0.4-linux-amd64.tar.gz $ tar xzvf etcd-v3.0.4-linux-amd64.tar.gz Add /home/mike/etcd-v3.0.4-linux-amd64 to your path in your ~/.profile In a first terminal run:

$ etcd

In a second terminal:

$ etcdctl set mykey "this is a test" $ etcdctl get mykey

Forking

Create your development forks on github for both

https://github.com/kubernetes/kubernetes

and

https://github.com/kubernetes/kubernetes.github.io

fork

Cloning your Forks

Normally, when cloning a github.com source repository, one uses the path $GOPATH/src/(path to source in github). This is for convenience when using go get type commands to pull down github hosted packages. However, for legacy reasons, Kubernetes packages are named k8s.io/kubernetes not github.com/kubernetes/kubernetes as you might expect. Thus, to make all of the development tooling work right and be able to find your package code on disk you’re going to have to put the clone in a k8s.io directory instead. Create the k8s.io/kubernetes source trees from your name/kubernetes forks using the following steps:

mike@mike-VirtualBox:~/go/src$ mkdir k8s.io mike@mike-VirtualBox:~/go/src$ cd k8s.io $ git clone https://github.com/mikebrow/kubernetes.git $ git clone https://github.com/mikebrow/kubernetes.github.io.git

Result: source files are copied (cloned) from your forks into the go/src/k8s.io/kubernetes and k8s.io/kubernetes.github.io.git directories. Git creates a master branch on your local hard disk for your git hub forks.

Check status of your branch:

$ cd kubernetes $ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working directory clean

Github Profile Recommendations

  • Add an email to your public profile
  • Enable two factor authentication

Git Client Configuration Recommendations

Set user.* entries to global and/or local:

$ git config --local user.name "Full name here" $ git config --local user.email "email address here" Or: $ git config --global user.name "Full name here" $ git config --global user.email "email address here"

Make it easier to push:

$ git config --global push.default simple

So you don’t have to remember your github credentials each time:

$ git config credential.helper store

Add a remote named upstream for fetching / pushing changes between the cloned branch on your hard disk and kubernetes/kubernetes:

$ git remote add upstream https://github.com/kubernetes/kubernetes.git

Running git config -l should look like this:

user.name=Mike Brown user.email=brownwm@us.ibm.com push.default=simple core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=https://github.com/mikebrow/kubernetes.git remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master remote.upstream.url=https://github.com/kubernetes/kubernetes.git remote.upstream.fetch=+refs/heads/*:refs/remotes/upstream/* credential.helper=store

The following steps are recommended to be run by the Kubernetes team. Note: The following link adds a very large bucket of build & verify commands to git commits which will be subsequently run when you commit changes against your branch.

$ cd .git/hooks/ $ ln -s ../../hooks/pre-commit .

If you don’t link the pre-commit checks you risk pushing a PR that will not pass muster. For example, if you modified docs on the k8s.io/kubernetes fork, and you did not run the corresponding documentation scripts, you risk checking in improperly modified docs.

Git Workflow

The following pic, shows a typical git workflow for finding and fixing an issue with an open source product on GitHub. While this flow was drafted for Docker, it also works for Kubernetes.

workflow

Find something to work on here: https://github.com/kubernetes/kubernetes/issues

findsomething

Make a comment on the issue you found.
  • If you have questions don’t be shy.
  • If you want to work on it, say so.
Call #dibs
  • Adding a #dibs comment to the issue tells others that you are going to work on a resolution to the issue. Calling dibs is a colloquialism meaning others should recognize that you asked to work this issue first. It is the equivalent of raising your hand first to a question given to an audience.
  • If you’re not sure whether you can fix it yourself… at least announce that you are investigating with a #investigating comment. You won’t be assigned and it won’t indicate that you have a resolution, but it lets others know that you might call dibs or open up a pull request later on.

Building Kubernetes

To build binaries for running locally:

$ sudo make

e2e Test note: With the default e2e test options, when the version of the client and server are out of sync the e2e tests won’t run. New: You can now set check_version_skew=false, to run the e2e with the versions out of skew. The client/server may not always build depending on what you changed. To force both to be in sync (clean/remove build binaries) with:

$ sudo make clean

To generate a release:

$ make release

To generate a release without running tests:

$ make release-skip-tests

Testing Kubernetes (unit and integration)

Unit tests:

$ sudo make test

Run just the pkg/api tests:

mike@mike-VirtualBox:~/go/src/k8s.io/kubernetes$ sudo WHAT=../../k8s.io/kubernetes/pkg/api make test

Integration Tests:

$ sudo make test-integration

Kubernetes e2e Tests

Running the Kubernetes e2e tests requires using a cluster. The e2e tests can be run on any Kubernetes compatible cluster. This guide shows you how to create a single node cluster running in your Linux environment. If you followed the above instructions, this cluster will be running in your Ubuntu guest in your VM. We will use two terminals. In terminal 1, build Kubernetes:

$ sudo make clean $ sudo make

Before running a local cluster make sure you have cfssl installed:

$ go get -u github.com/cloudflare/cfssl/cmd/...

Bring up your local cluster:

$ sudo PATH=$PATH KUBERNETES_PROVIDER=local hack/local-up-cluster.sh

In terminal 2, configure kubectl to interact with your local cluster:

$ export KUBERNETES_PROVIDER=local $ cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/apiserver.crt $ cluster/kubectl.sh config set-credentials myself --username=admin --password=admin $ cluster/kubectl.sh config set-context local --cluster=local --user=myself $ cluster/kubectl.sh config use-context local

Check status of your single node native local cluster:

$ cluster/kubectl.sh get nodes NAME STATUS AGE VERSION 127.0.0.1 Ready 10m v1.6.0-alpha.0.2912+de59ede6b2f4c2-dirty

To configure for running e2e tests on your single node local cluster add the following to your ~/.profile:

export KUBECONFIG=/home/mike/.kube/config

Check your environment:

mike@mike-VirtualBox:~/go/src/k8s.io/kubernetes$ env | grep kube

KUBECONFIG=/home/mike/.kube/config PATH=/home/mike/Downloads/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/mike/go/bin:/snap/bin:/home/mike/go/bin:/home/mike/.rvm/bin:/home/mike/go/src/github.com/kubernetes/kubernetes/third_party/etcd:/home/mike/.rvm/bin PWD=/home/mike/go/src/k8s.io/kubernetes

To run the e2e tests:

$ KUBE_MASTER_IP=localhost:6443 KUBE_MASTER=https://localhost:6443 go run hack/e2e.go -v --test --test_args="--host=https://localhost:6443"

To run just the e2e Conformance tests:

$ KUBE_MASTER_IP=localhost:6443 KUBE_MASTER=https://localhost:6443 go run hack/e2e.go -v --test --test_args="--host=https://localhost:6443 --ginkgo.focus=through.*Conformance"

When finished using your Kubernetes cluster you can quickly bring it down by hitting ctrl-c in terminal 1.

Running Documentation related scripts

After editing documentation related files in the kubernetes/kubernetes fork use before doing a git commit:

$ sudo hack/update-all.sh note: requires godep to be installed for sudo as well as $user

Alternatively you can call the following or do a commit and find out which of the following you need to run:

$ sudo hack/update-generated-docs.sh $ sudo hack/update-generated-swagger-docs.sh $ sudo hack/update-swagger-spec.sh

kubernetes.github.io Documents

Many of the kubernetes md files (documentation files) have been moved from the kubernetes/kubernetes repository to the kubernetes/kubernetes.github.io repository. For example, the documents published at the documentation web site http://kubernetes.io/docs/ are generated from the kubernetes/kubernetes.github.io repository. If you followed the earlier instructions you should have a directory named ~/go/src/k8s.io/kubernetes.github.io where these files are located. Follow these instructions to build and host a local version of the kubernetes documentation website:

http://kubernetes.io/editdocs/

In this fashion you can edit the docs and get a preview of what the changes will look like up on kubertnetes.io.

Guidelines/Things to do:

  • Run gofmt -s -w file.go on each changed file before committing your changes.
  • Run golint on each changed file before committing your changes. (Kubernetes team does not currently use lint. But it doesn’t hurt.)
  • Remember to update the documentation when creating or modifying features.
  • Remember to add test cases to cover your changes.
  • Commits that fix or close an issue should reference them in the commit message Closes #XXXX or Fixes #XXXX.
  • After every commit, run the test suite and ensure it is passing.
  • Sync and rebase frequently as you code to keep up with master.
  I hope this guide helps you get started with your development for Kubernetes. Cheers, Mike Brown

Contributions

I’d like to give a special thanks to Douglas Davis and Morgan Bauer for their help in putting this guide together.

7 Comments on "Kubernetes Developer Guide – Part 1"

  1. […] Kubernetes Developer Guide – Part 1 […]

  2. Satish Chhatpar June 30, 2016

    Thanks for sharing this knowledge Mike
    I will try this

  3. Thanks for sharing.
    This is exactly what I need to know more Kube

  4. Nice sharing, Mike, thanks

  5. Thanks Xing good document. Its worked for me .
    can you share the kubernetes development setup with any IDE, Thanks.

Join The Discussion

Your email address will not be published. Required fields are marked *