Digital Developer Conference: Cloud Security 2021 -- Build the skills to secure your cloud and data Register free

Use Kubernetes service accounts to enable automated kubectl access

With service accounts, you can connect to the Kubernetes API from inside pods running in a cluster. But what if you have an external script that uses kubectl, oc, or a client library and you want to connect to the API from outside in a way that is not tied to any particular user? This tutorial shows you how to effectively build a KUBECONFIG file for this purpose.

Normally, you would access your Kubernetes or Red Hat OpenShift cluster from the command line by using kubectl or oc, and a corresponding KUBECONFIG file is created (and occasionally updated). This process happens automatically without any substantial user action. However, there might be times when you need to access the cluster from programs using the API that runs outside the cluster. This can be simple scripts (such as in Bash) that call out to the kubectl or oc commands, or proper client applications that use language-specific clients such as the Kubernetes Go client. While it is obvious that KUBECONFIG files are needed for the former, even the official Kubernetes instructions for external access using the Go client begin by showing you how to load a KUBECONFIG file.

So why not just copy-paste and use the KUBECONFIG file that’s on the machine you would normally use to access the cluster?

There are a couple of reasons:

  1. The script is now tied to your user, which is, in general, not considered to be a clean design. In addition, you might be violating the Principle of Least Privilege, which allows for general-purpose access to a script that most likely does not need that much power.
  2. If you log in to your cluster by using a command such as oc login, then cluster access is configured typically through OAuth. This means that a temporary token is fetched and the KUBECONFIG file is rewritten every time that you log in. Even if you don’t care about the first reason, you will be physically unable to use this file as it will stop working when either the token expires after 24 hours or the next oc login or oc logout command is issued.

Clearly, you need a cleaner solution. Specifically, a KUBECONFIG that is not tied to a user, has the right set of privileges, and works permanently. Service accounts can be used for this purpose, but this process is not clearly documented. For example, the Obtaining the service account token by using kubectl section of the IBM Cloud Pak for Multicloud Management product documentation shows that this is possible, but it does not provide sufficient detail.

This tutorial walks you through the process of creating a KUBECONFIG file that can access and create pods in a particular namespace in your cluster. Specifically, a namespace that is not tied to a specific user and works permanently. Any external script can now use this KUBECONFIG file for kubectl commands or for clients in other languages.

Prerequisites

To complete this tutorial, you need a basic knowledge of kubectl commands and a Kubernetes or OpenShift cluster.

Estimated time

It should take you about 15 minutes to complete this tutorial.

Steps

To complete this tutorial, you need to:

  1. Set up your service account
  2. Extract the token from the service account
  3. Create the KUBECONFIG file

Step 1. Set up your service account

Service accounts are the official way to access the Kubernetes API from within pods, and there are several tutorials that cover this well, such as the Configure Service Accounts for Pods tutorial within the Kubernetes documentation. However, tutorials such as this do not explain the fact that a service account secret can be used from scripts outside a pod, or even outside a cluster, such as a script that calls kubectl.

Due to the advanced role-based access control (RBAC) system in Kubernetes, not all service accounts are the same. For this tutorial, let’s create a fresh one.

Note: You are free to reuse an existing account, but as the following steps demonstrate, there is no guarantee that you will be able to find one with the access control rights that you need. There are two possible results: Either the service account that you use will not be able to do what you want, or it will end up with far more rights than the calling script requires, which from a security standpoint is not ideal according to the Principle of Least Privilege.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myexample-sa
  namespace: myexample

Note: All resources similar to the one shown here must be applied into your cluster. Save the text in a YAML file and use kubectl apply -f <filename>.

After you have the service account, you need to create a Role that represents the access rights that you want for your script and bind it to this service account.

First, the Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: myexample
  name: myexample-role
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list", "create"]
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods/exec"]
  verbs: ["create"]

You could use ClusterRole instead of Role, which enables you to work across namespaces, but we will stick with Role here. The Using RBAC Authorization section of the Kubernetes documentation provides details on how to configure this resource, but in this example we want to allow this Role to use the verbs get, watch, list, and create on all pods.

The normal list of verbs is get, watch, list, create, update, patch, and delete. The apiGroups are of the form app, batch, extensions, and so on, which you might recognize as the prefix in the apiVersion used in the resource specifications. You can obtain the list of resources by using the kubectl api-resources -o wide command, which also lists verbs that are relevant to each resource.

Note that you must include the create verb for a resource called pods/exec to give your script the ability to kubectl exec into the pods. These resources are known as subresources. The other relevant subresource for pods is pods/log, which enables you to run the kubectl logs command against the pod. Unfortunately, direct documentation for all of the available subresources is not available. Another resource that does not appear to be listed is events, which opens up the list of events in kubectl describe. Without this permission, events are not listed.

One easy way to get a full list of resources and subresources that are available on a cluster is to look at the ~/.kube/cache/discovery/<cluster name> folder. Each subfolder here refers to an apiGroup, such as batch or apps. Each subfolder has a v1 subfolder with a file called serverresources.json, which contains the list of resources and subresources with the relevant verbs. The advantage of this approach is that you get a listing that is relevant to your cluster, which includes any custom resources that you installed in that cluster.

Also note that ClusterRoles are necessary to access cluster-wide resources such as nodes.

Here’s how to bind the myexample-role role to the myexample-sa service account:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myexample-rb
  namespace: myexample
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: myexample-role
subjects:
- kind: ServiceAccount
  name: myexample-sa
  namespace: myexample

After you create these resources on the cluster, the service account can do what you want.

Note: As before, if you have ClusterRoles, you should create corresponding ClusterRoleBindings, which look exactly like the RoleBinding to the same subject, myexample-sa. You can create any number of RoleBindings and ClusterRoleBindings for a single subject.

Step 2. Extract the token from the service account

Now that you have a service account, it can be used from the pods. However, we are more interested in the secret token that it holds, so let’s extract that out.

First, take a look at the service account you just created:

$ kubectl -n myexample describe sa myexample-sa

The output should look similar to the following:

    Name:                myexample-sa
    Namespace:           myexample
    Labels:              <none>
    Annotations:         Image pull secrets:  myexample-sa-dockercfg-xxxxx
    Mountable secrets:   myexample-sa-token-xxxxx
                         myexample-sa-dockercfg-xxxxx
    Tokens:              myexample-sa-token-xxxxx
                         myexample-sa-token-xxxxx
    Events:              <none>

The thing we are most interested in is the token, so you should choose one of the two available options.

Now, look at the secret it represents:

$ kubectl describe secret myexample-sa-token-xxxxx

It should look similar to the following:

        Name:         myexample-sa-token-xxxxx
        Namespace:    myexample
        Labels:       <none>
        Annotations:  kubernetes.io/created-by: <...>
                      kubernetes.io/service-account.name: myexample-sa
                      kubernetes.io/service-account.uid: <...>
        Type:  kubernetes.io/service-account-token
        Data
        ====
        ca.crt:          5940 bytes
        namespace:       9 bytes
        service-ca.crt:  8365 bytes
        token:
        eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJmaXZlZy1jb3JlLWlybCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJmaXZlZ2MtYWNjZXNzLXNhLXRva2VuLWNkcmduIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImZpdmVnYy1hY2Nlc3Mtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3OGUxMDE0OS0wZDIzLTRhNDctYTdmMS00YmFiMzNlMWIxODciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Zml2ZWctY29yZS1pcmw6Zml2ZWdjLWFjY2Vzcy1zYSJ9.LnAFgWZdTpzBpiDYQpgqZDwhWBVpqrq97uOzp7qTAWrp825g60uu.9ic9Kz9EqhLUhl_vE-eyJhbGciOiJSUzI1NiIsImtpZCI6IkRhVFYzT3l0UEZrbmN0YkFFUlRfRkhGUUt5TU8tYV80V1Ayc1VXc3pma1kifQ-nSXW3_26sQdXnIA-sTzujFPROvVmWKWJNR0_Y8B2z-MsOH2IGVX9jeiyYByqfACXF83DepVLsJLOQKUbxcXVpjWtKpR4GpWs6pGxiR3ufzMA5BVE2Rgw0e4g_L8zBLDn36SdYFkW_S7wh8-fOKLhuNk1O60afTuZbf4cnzNwZyYgcZBTmrfJQHRlMRS3r2Nz_BXGQWX9VGNVCEvE_vfCxoOxWECeJ6QIbTf9dYcYlde8clVcHxdZKZOmex-m3-oxZKZJdE0vA4QOMBb

The token field is of most interest here. Of course, the token shown here is just a randomized string. Make a note of this.

Note: If you use get -o yaml instead of describe, you will get a base64-encoded version of the token, and you must manually decode it before you can use it in the next step.

At this point, you have a legitimate API key to access the Kubernetes API from anywhere. If you access the API directly (say, from curl), you can use this token as the bearer token for the authorization header.

However, Kubernetes clients usually make you load configs from the KUBECONFIG file. For instance, the client-go example within the Kubernetes project repo. Therefore, you need to prepare one, as the next step demonstrates.

Step 3. Create the KUBECONFIG file

Normally, if you are using multiple clusters, the KUBECONFIG file can be a bit of a mess. So it is better to create a new file for this purpose.

The structure of the file should look similar to the following:

        apiVersion: v1
        clusters:
        - cluster:
            insecure-skip-tls-verify: true
            server: https://<url>:6443
          name: <url>:6443
        contexts:
        - context:
            cluster: <url>:6443
            namespace: myexample
            user: myexample-sa
          name: default
        current-context: default
        kind: Config
        preferences: {}
        users:
        - name: myexample-sa
          user:
            token: <your-token>

Let’s go over this section by section:

  1. In the clusters section, you can use the values from your main KUBECONFIG file or the URL directly if you know it. There is only one cluster here, but this technique can be repeated any number of times.
  2. In the contexts section, each context consists of a namespace and user in a cluster. In this example, you have only one namespace, myexample, and one user represented by myexample-sa. As before, you can repeat the previous steps for any number of namespaces and users.
  3. The current-context is set to whichever of the configs you want to apply by default.
  4. In the users section, you have a user standing in for your service account. This is the main location of change: You need to fill in the token that was extracted from the service account secret in the previous section here.

That’s it. After you have this file, running your scripts should be as simple as:

KUBECONFIG=<path to new kubeconfig> myscript

If you have multiple contexts, say for doing things in different namespaces, you can just switch context. For example, with something similar to the following inside of myscript:

        <do something in default context>

        kubectl config use-context context1
        <do something in default context1>

        kubectl config use-context context2
        <do something in default context2>

        # reset context
        kubectl config use-context default

Summary

You created a fresh service account with desired access controls, extracted the token secret from it, and used it to create a KUBECONFIG file. You can now use this file from anywhere to allow programmatic client access in a way that is safe and permanent.

Now that you have a KUBECONFIG file, you can use the kubectl cheat sheet from Kubernetes documentation as a handy kubectl command reference. Also, the Kubernetes API reference can help you write resource config files and interact with your cluster.