Tutorial

Archived | Using Kubernetes RBAC and service accounts

Learn how to expose parts of the Kubernetes API with RBAC

By Sean Dague
Published July 26, 2018

Archived content

Archive date: 2019-10-24

This content is no longer being updated or maintained. The content is provided “as is.” Given the rapid evolution of technology, some content, steps, or illustrations may have changed.

One of the powerful aspects of Kubernetes is the ability for applications to call the Kubernetes API for advanced configuration. Starting in Kubernetes 1.8, access to the API was put under a Role Based Access Control model for increased security. We’ll take some time to look at how this changes by using the API in Kubernetes and how to build configuration that correctly uses RBAC.

Learning Objectives

Upon completing this tutorial, you will understand how to:

Expose parts of the Kubernetes API using Roles and RoleBindings.
Create a ServiceAccount to further restrict which pods can make API calls.

Prerequisites

In order to complete this tutorial, you will need the following prerequisites:

An IBM Cloud account with the ability to create a Kubernetes cluster. Sign up if you don’t have an account yet.
A provisioned Kubernetes cluster in the IBM Cloud Kubernetes Service.
Install the IBM Cloud Developer Tools.
Git Clone the rbac-in-k8s git repository – this provides all the configuration files for creating images and the Kubernetes yaml files that are required for walking through this tutorial.

Estimated Time

The total time to complete this tutorial is around 90 minutes.

Steps

1. Build application images

Let’s build some images for our application and get them up and running. These are modified versions of the images used in the ny-power application. They include the kubectl tool to make it easy to access the Kubernetes API.

First off, we need to create a container registry namespace to store our images:

> ibmcloud cr namespace-add rbac-tutorial

Adding namespace 'rbac-tutorial'...

Successfully added namespace 'rbac-tutorial'

OK

Then we’ll build these example images:

> ibmcloud cr build --tag registry.ng.bluemix.net/rbac-tutorial/mqtt-img:1 deploy/mqtt-img

Sending build context to Docker daemon  6.656kB
Step 1/13 : FROM ubuntu:xenial
...
1: digest: sha256:ea1afeb4e5754f8defcae039f9a43aff8b81ecc24ef9ed2d907381a9a99d0b2b size: 2821

OK

> ibmcloud cr build --tag registry.ng.bluemix.net/rbac-tutorial/tools-img:1 deploy/tools-img

Sending build context to Docker daemon  4.096kB
Step 1/9 : FROM ubuntu:xenial
...
1: digest: sha256:78b0639d6c77af5b6fda4d690273a702c43c55cb386f5ab24e78a693a6664f2a size: 2200

OK

2. Start our sample application and our tools pod

We’re going to start up two deployments based on the built images from step 1:

> kubectl apply -f deploy/mqtt.yaml -f deploy/tools.yaml

secret "mqtt-secret" created
persistentvolumeclaim "mqtt-nfs" created
deployment.apps "mqtt" created
service "mqtt" created
deployment.apps "tools-no-rbac" created

This is going to take a 2 – 3 minutes to provision. kubectl get all will display the current status. When both pods are in the Running state you can proceed to the next steps.

> kubectl get all
NAME                                 READY     STATUS    RESTARTS   AGE
pod/mqtt-5ccf8b68b6-m8hfl            1/1       Running   0          2m
pod/tools-no-rbac-7dc96f489b-d9gcl   1/1       Running   0          2m

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                       AGE
service/kubernetes   ClusterIP      172.21.0.1       <none>          443/TCP                       16d
service/mqtt         LoadBalancer   172.21.173.243   169.60.93.179   1883:31532/TCP,80:31517/TCP   2m

NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/mqtt            1         1         1            1           2m
deployment.extensions/tools-no-rbac   1         1         1            1           2m

NAME                                             DESIRED   CURRENT   READY     AGE
replicaset.extensions/mqtt-5ccf8b68b6            1         1         1         2m
replicaset.extensions/tools-no-rbac-7dc96f489b   1         1         1         2m

NAME                            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mqtt            1         1         1            1           2m
deployment.apps/tools-no-rbac   1         1         1            1           2m

NAME                                       DESIRED   CURRENT   READY     AGE
replicaset.apps/mqtt-5ccf8b68b6            1         1         1         2m
replicaset.apps/tools-no-rbac-7dc96f489b   1         1         1         2m

3. Attempt to access the Kubernetes API

One of the powerful parts of Kubernetes is the ability to interact with the Kubernetes API inside the cluster. This allows applications to actively manage their own resources and adapt to circumstances.

We’ll demonstrate that by connecting to our tools app and using kubectl.

Run the following command to get the name of the tools pod that we launched in the environment.

> kubectl get pods -l rbac=none
NAME                             READY     STATUS    RESTARTS   AGE
tools-no-rbac-7dc96f489b-ph7h9   1/1       Running   0          1h

Now we can create a bash session on that pod by using the following command:

> kubectl exec -it tools-no-rbac-7dc96f489b-ph7h9 bash

root@tools-no-rbac-7dc96f489b-ph7h9:/#

We’re in! The next step is to run kubectl get all to see what resources we can see from there.

root@tools-no-rbac-7dc96f489b-ph7h9:/# kubectl get all

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:default" cannot list pods in the namespace "default"
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:default:default" cannot list replicationcontrollers in the namespace "default"
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list services in the namespace "default"
Error from server (Forbidden): daemonsets.extensions is forbidden: User "system:serviceaccount:default:default" cannot list daemonsets.extensions in the namespace "default"
Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:default:default" cannot list deployments.extensions in the namespace "default"
Error from server (Forbidden): replicasets.extensions is forbidden: User "system:serviceaccount:default:default" cannot list replicasets.extensions in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:default:default" cannot list daemonsets.apps in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list deployments.apps in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:default:default" cannot list replicasets.apps in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:default:default" cannot list statefulsets.apps in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:default:default" cannot list horizontalpodautoscalers.autoscaling in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:default:default" cannot list jobs.batch in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:default:default" cannot list cronjobs.batch in the namespace "default"

Okay that didn’t work. So what happened?

Starting in Kubernetes 1.9, the API was put behind a mandatory Role Based Access Control system. By default, access is no longer granted to applications. You must now explicitly allow access to the parts of the API that your applications need. This broke a lot of applications that weren’t prepared for the transition.

Kubernetes has two resources that control the access to the API:

Role: specifies what access is granted
RoleBinding: specifies who the Role applies to

We’ll create both in a few ways to see how this all works.

4. Create a Role and RoleBinding

Our first step is to create a Role. The example Role can do two things: list or get all services, and create or delete secrets.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: api-role
  namespace: default
  labels:
    app: tools-rbac
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["mqtt-pub-address"]
  verbs: ["update", "delete"]

Roles are specified as a set of rules. They are based on the apiGroup (empty for core resources), resource name, verbs to act on the that resource, and optionally a resourceName to restrict it further (often used with secrets and configmaps).

This Role will let us list the services, get information on a particular service, and create or delete a single secret.

A Role in isolation doesn’t do anything until we bind it with a RoleBinding.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: global-rolebinding
  namespace: default
  labels:
    app: tools-rbac
subjects:
- kind: Group
  name: system:serviceaccounts
  apiGroup: rbac.authorization.k8s.io
  namespace: default
roleRef:
  kind: Role
  name: api-role
  apiGroup: ""

A RoleBinding links a Role to Subjects. There are many different ways to handle Subjects. In this case, we’ll give this Role to all service accounts in the default namespace. This effectively means that all pods will have access to these APIs.

This can be applied with the yaml file in the repository:

> kubectl apply -f deploy/role.yaml -f deploy/global-role-assign.yaml
role.rbac.authorization.k8s.io "api-role" created
rolebinding.rbac.authorization.k8s.io "global-rolebinding" created

5. Test our new Access

Let’s connect to our tools pod and see what happens now:

> kubectl exec -it tools-no-rbac-7dc96f489b-ph7h9 bash

root@tools-no-rbac-7dc96f489b-ph7h9:/# kubectl get all
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                       AGE
kubernetes   ClusterIP      172.21.0.1     <none>          443/TCP                       15d
mqtt         LoadBalancer   172.21.91.88   169.60.93.179   1883:32145/TCP,80:31639/TCP   22h
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:default" cannot list pods in the namespace "default"
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:default:default" cannot list replicationcontrollers in the namespace "default"
Error from server (Forbidden): daemonsets.extensions is forbidden: User "system:serviceaccount:default:default" cannot list daemonsets.extensions in the namespace "default"
Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:default:default" cannot list deployments.extensions in the namespace "default"
Error from server (Forbidden): replicasets.extensions is forbidden: User "system:serviceaccount:default:default" cannot list replicasets.extensions in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:default:default" cannot list daemonsets.apps in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list deployments.apps in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:default:default" cannot list replicasets.apps in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:default:default" cannot list statefulsets.apps in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:default:default" cannot list horizontalpodautoscalers.autoscaling in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:default:default" cannot list jobs.batch in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:default:default" cannot list cronjobs.batch in the namespace "default"

root@tools-no-rbac-7dc96f489b-ph7h9:/# kubectl get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                       AGE
kubernetes   ClusterIP      172.21.0.1     <none>          443/TCP                       15d
mqtt         LoadBalancer   172.21.91.88   169.60.93.179   1883:32145/TCP,80:31639/TCP   22h

We can see that we now have access to services in the cluster.

The next thing we want to do is create a configmap entry to our mqtt public address. We can do that with:

root@tools-no-rbac-7dc96f489b-ph7h9:/# kubectl create configmap mqtt-pub-address --from-literal=host=169.60.93.179
configmap "mqtt-pub-address" created

After it’s created from within the pod we can’t get it (because we didn’t provide that level of access).

root@tools-no-rbac-7dc96f489b-ph7h9:/# kubectl get configmap/mqtt-pub-address
Error from server (Forbidden): configmaps "mqtt-pub-address" is forbidden: User "system:serviceaccount:default:default" cannot get configmaps in the namespace "default"

If we look instead at this from our computer where we have all the permissions, we can see the contents of that configmap.

> kubectl get configmap/mqtt-pub-address -o yaml
apiVersion: v1
data:
  host: 169.60.93.179
kind: ConfigMap
metadata:
  creationTimestamp: 2018-07-12T14:45:13Z
  name: mqtt-pub-address
  namespace: default
  resourceVersion: "418889"
  selfLink: /api/v1/namespaces/default/configmaps/mqtt-pub-address
  uid: 2eee2331-85e2-11e8-857f-06cd14ab6bce

We can also see that we gave this access to every pod in our environment. If we connect to our mqtt pod we can run the same command:

> kubectl get pod -l app=mqtt
NAME                    READY     STATUS    RESTARTS   AGE
mqtt-5ccf8b68b6-bkdf9   1/1       Running   0          2m

> kubectl exec -it mqtt-5ccf8b68b6-bkdf9 bash

root@mqtt-5ccf8b68b6-bkdf9:/# kubectl get services

NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                       AGE
kubernetes   ClusterIP      172.21.0.1     <none>          443/TCP                       15d
mqtt         LoadBalancer   172.21.91.88   169.60.93.179   1883:32145/TCP,80:31639/TCP   23h

That is way more access than we wanted to grant. Let’s see what we can do about granting more specific access to just the tools pod.

Before we do that, we’ll need to remove the global-role-binding so that it doesn’t get in the way of future examples.

> kubectl delete -f deploy/global-role-assign.yaml

rolebinding.rbac.authorization.k8s.io "global-rolebinding" deleted

And we can see that our access has been revoked for the tools pod:

root@tools-no-rbac-7dc96f489b-ph7h9:/# kubectl get services

Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list services in the namespace "default"

Recap: What we learned so far

From this example we learned the following things:

By default, access to the Kubernetes API is restricted in a cluster.
We can grant access using Role and RoleBinding resources.
Roles have a set of Rules based on resource, verbs, and sometimes resourceNames.
Binding to the Subject of kind: Group and name: system:serviceaccounts will give access to all pods in the system.

6. Create a Service Account

The best practice in security is to give out as few permissions as possible. One way to do that in Kubernetes is through ServiceAccounts. By default, all applications run under a default ServiceAccount, but we can create additional ones that are specific to our application.

The following yaml defines a basic ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: service-account-1
  labels:
    app: tools-rbac

We can start a pod with a ServiceAccount by adding that to its spec definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tools-service-account
  labels:
    app: tools
    rbac: service-account-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tools
      rbac: service-account-1
  template:
    metadata:
      labels:
        app: tools
        rbac: service-account-1
    spec:
      serviceAccountName: service-account-1
      containers:
        - name: tools
          image: "registry.ng.bluemix.net/rbac-tutorial/tools-img:1"
          imagePullPolicy: Always
          command: ["/bin/sleep", "3601"]

In the pod spec you can see serviceAccountName: service-account-1. The pod will be run as this service account, and all containers started from it will be running under that service account.

7. Start the Deployment with this ServiceAccount

Run the following to start this pod with the service account in question:

> kubectl apply -f deploy/tools-service-account.yaml

serviceaccount "service-account-1" configured
deployment.apps "tools-service-account" configured

Great, now lets see how our pod is doing:

> kubectl get pods
NAME                                     READY     STATUS         RESTARTS   AGE
mqtt-5ccf8b68b6-bkdf9                    1/1       Running        0          51m
tools-no-rbac-7dc96f489b-ph7h9           1/1       Running        22         22h
tools-service-account-6664bdf7f-jzpcg    0/1       ErrImagePull   0          36s

Hmmm… that’s no good, why didn’t our pod start?

> kubectl describe pod/tools-service-account-6664bdf7f-jzpcg

...
Events:
  Type     Reason                 Age               From                     Message
  ‑‑‑‑     ‑‑‑‑‑‑                 ‑‑‑‑              ‑‑‑‑                     ‑‑‑‑‑‑‑
  Normal   Scheduled              2m                default-scheduler        Successfully assigned tools-service-account-6664bdf7f-jzpcg to 10.188.103.254
  Normal   SuccessfulMountVolume  2m                kubelet, 10.188.103.254  MountVolume.SetUp succeeded for volume "service-account-1-token-kcbfz"
  Normal   Pulling                1m (x4 over 2m)   kubelet, 10.188.103.254  pulling image "registry.ng.bluemix.net/rbac-tutorial/tools-img:1"
  Warning  Failed                 1m (x4 over 2m)   kubelet, 10.188.103.254  Failed to pull image "registry.ng.bluemix.net/rbac-tutorial/tools-img:1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.ng.bluemix.net/v2/rbac-tutorial/tools-img/manifests/1: unauthorized: authentication required
  Warning  Failed                 1m (x4 over 2m)   kubelet, 10.188.103.254  Error: ErrImagePull
  Normal   BackOff                48s (x6 over 2m)  kubelet, 10.188.103.254  Back-off pulling image "registry.ng.bluemix.net/rbac-tutorial/tools-img:1"
  Warning  Failed                 48s (x6 over 2m)  kubelet, 10.188.103.254  Error: ImagePullBackOff

It appears that our new service account doesn’t have access to our image registry. If we do a get on both the default service account and service-account-1 we can see a critical difference.

> kubectl get sa default -o yaml

apiVersion: v1
imagePullSecrets:
- name: bluemix-default-secret
- name: bluemix-default-secret-regional
- name: bluemix-default-secret-international
kind: ServiceAccount
metadata:
  creationTimestamp: 2018-06-26T20:33:40Z
  name: default
  namespace: default
  resourceVersion: "241"
  selfLink: /api/v1/namespaces/default/serviceaccounts/default
  uid: 360775a5-7980-11e8-857f-06cd14ab6bce
secrets:
- name: default-token-x5gbt

> kubectl get sa service-account-1 -o yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"labels":{"app":"tools-rbac"},"name":"service-account-1","namespace":"default"}}
  creationTimestamp: 2018-07-11T18:05:35Z
  labels:
    app: tools-rbac
  name: service-account-1
  namespace: default
  resourceVersion: "420289"
  selfLink: /api/v1/namespaces/default/serviceaccounts/service-account-1
  uid: 02a91878-8535-11e8-857f-06cd14ab6bce
secrets:
- name: service-account-1-token-kcbfz

The default service account has this additional set of attributes under imagePullSecrets. These are what enable the service accounts access to the image registry. If we update our service account definition to include these our image should come up.

> kubectl apply -f deploy/fix-service-account-1.yaml

serviceaccount "service-account-1" configured

> kubectl delete pod/tools-service-account-6664bdf7f-jzpcg

pod "tools-service-account-6664bdf7f-jzpcg" deleted

We delete the old pod that was in a failed state because in this error condition, Kubernetes will not automatically retry launching it. After a delete of the pod, the deployment will try to start the pod again and this time it will succeed.

> kubectl get pods
NAME                                    READY     STATUS    RESTARTS   AGE
mqtt-5ccf8b68b6-bkdf9                   1/1       Running   0          1h
tools-no-rbac-7dc96f489b-ph7h9          1/1       Running   22         22h
tools-service-account-6664bdf7f-rv5n2   1/1       Running   0          1m

8. Add Role and RoleBinding for Service Account

We now are running the tools-service-account pod as service-account-1. The following configuration will bind our previous Role to just this service account.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-account-rolebinding
  namespace: default
  labels:
    app: tools-rbac
subjects:
- kind: ServiceAccount
  name: service-account-1
roleRef:
  kind: Role
  name: api-role
  apiGroup: ""

The difference from our global assignment is that our subject is now of kind: ServiceAccount instead of kind: Group, which exposes this to only the single service account.

We can apply these changes with:

> kubectl apply -f deploy/service-account-role-assign.yaml

rolebinding.rbac.authorization.k8s.io "service-account-rolebinding" configured

9. Test access

Let’s see the impact of these changes in our pods. We’ll just exec the commands directly in all running pods.

> kubectl get pods
NAME                                    READY     STATUS    RESTARTS   AGE
mqtt-5ccf8b68b6-bkdf9                   1/1       Running   0          5h
tools-no-rbac-7dc96f489b-ph7h9          1/1       Running   26         1d
tools-service-account-6664bdf7f-rv5n2   1/1       Running   3          3h

> kubectl exec mqtt-5ccf8b68b6-bkdf9 kubectl get services
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list services in the namespace "default"
command terminated with exit code 1

> kubectl exec tools-no-rbac-7dc96f489b-ph7h9 kubectl get services
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list services in the namespace "default"
command terminated with exit code 1

> kubectl exec tools-service-account-6664bdf7f-rv5n2 kubectl get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                       AGE
kubernetes   ClusterIP      172.21.0.1     <none>          443/TCP                       15d
mqtt         LoadBalancer   172.21.91.88   169.60.93.179   1883:32145/TCP,80:31639/TCP   1d

This is much better. As expected, the only pod that has access to make these API calls is the one that is running as service-account-1. The others haven’t been granted any access.

Recap – What we learned so far

From this example we learned the following things:

We can create additional ServiceAccounts besides the default to isolate pods.
ServiceAccounts may need explicit permissions to access private image registries.
We can assign Roles to specific ServiceAccounts with RoleBindings to ensure only specific pods have access to the API

10. ClusterRoles and ClusterRoleBinding

Roles and RoleBindings only apply to a single namespace. That’s all we wanted thus far. But what if we wanted to provide access across all namespaces? Or what if we wanted to provide access to resources that don’t live in a namespace, like nodes or even the non-resource healthz endpoint? For that we need ClusterRoles and ClusterRoleBindings.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-cluster-role
rules:
‑ apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
‑‑‑
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-account-clusterrolebinding
  labels:
    app: tools-rbac
subjects:
‑ kind: ServiceAccount
  name: service-account-1
  namespace: default
roleRef:
  kind: ClusterRole
  name: node-cluster-role
  apiGroup: ""

This looks very similar to the namespace specific version, but one major difference is that when referencing subject we need to specify which namespace the service account is located. ClusterRoleBindings don’t live in a namespace, so we have no default to work with.

We can apply this with:

> kubectl apply -f deploy/service-account-cluster-role.yaml

clusterrole.rbac.authorization.k8s.io "node-cluster-role" configured
clusterrolebinding.rbac.authorization.k8s.io "service-account-clusterrolebinding" created

Now the following commands work:

> kubectl exec tools-service-account-6664bdf7f-rv5n2 kubectl get nodes

NAME             STATUS    ROLES     AGE       VERSION
10.188.103.209   Ready     <none>    15d       v1.9.8-2+af27ab4b096122
10.188.103.229   Ready     <none>    15d       v1.9.8-2+af27ab4b096122
10.188.103.254   Ready     <none>    15d       v1.9.8-2+af27ab4b096122

But as expected, they will fail from every other pod in the cluster, as we assigned it to a single ServiceAccount:

> kubectl exec  tools-no-rbac-7dc96f489b-ph7h9 kubectl get nodes

Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:default:default" cannot list nodes at the cluster scope
command terminated with exit code 1

A Real World Example – ny-power

Why might we want to access the internals like this? Here is the example that the tutorial was based on.

The ny-power application is an MQTT message streaming service that provides real time data from the New York State power grid. There is same web console that attaches directly to this MQTT public port. To do this, we need to pass the public ip address from the Load Balancer to the pod that starts the web site.

It’s also not a great idea to have things like kubectl living in all your images, so we created an init container that boots up, examines the services, extracts the public ip, and sets that in a secret. Once complete, the web container boots and pulls that public IP out of its environment, allowing the container to be much thinner with less access to tools.

A service account is used so that only this pod has any access to the API. Below is the example configuration from the application. You can also read more about the whole application here.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ny-power-web
    chart: ny-power-0.2.0
    heritage: Tiller
    release: prod
  name: prod-ny-power-web
  namespace: default
spec:
  selector:
    matchLabels:
      app: ny-power-web
      release: prod
  template:
    spec:
      serviceAccountName: prod-ny-power-readersa
      initContainers:
      ‑ command:
        ‑ /root/setvalue.sh
        env:
        ‑ name: MQTT_CONTAINER_NAME
          value: prod-ny-power-mqtt
        ‑ name: MQTT_SECRET_NAME
          value: prod-ny-power-mqtt
        image: registry.ng.bluemix.net/ny-power/ny-power-ibm-cloud:9
        imagePullPolicy: Always
        name: prod-ny-power-web-init
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      containers:
      ‑ env:
        ‑ name: INFLUXDB_HOST
          value: prod-ny-power-influx
        ‑ name: MQTT_HOST
          valueFrom:
            secretKeyRef:
              key: host
              name: prod-ny-power-mqtt
        ‑ name: MQTT_PUMP_PASS
          valueFrom:
            secretKeyRef:
              key: password
              name: prod-ny-power-mqtt-pump
        image: registry.ng.bluemix.net/ny-power/ny-power-web:17
        imagePullPolicy: Always
        name: prod-ny-power-web
        ports:
        ‑ containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always

Summary

In this tutorial, we learned:

The Kubernetes API is accessible inside the cluster.
By default, API access in Kuberenetes 1.8 and later is denied by Role Based Access Control.
You can enable specific permissions using Role and RoleBinding resources.
Role permissions on Secrets and Configmaps can include the name of the secret or configmap keys in question, providing very fine-grained access.
You can narrow the scope of permissions further by using a ServiceAccount for your applications, and RoleBinding to a ServiceAccount instead of to a group or user.
Some APIs in Kubernetes don’t have a pod scope (such as node access), and are exposed with the ClusterRole and ClusterRoleBinding.

Learn more

RBAC is a complicated subject and keeping the model of it in your head is sometimes challenging. Hopefully this tutorial and the ability to interactively poke at an example application helps clarify things.

If you are interested in learning more, check out the resources below.

Read the Kubernetes RBAC Docs – these are the definitive source for how RBAC works, and has other examples to show common usage.
Examine the code of the ny-power pattern to see this action.

Get Involved

Workshops