Overview

Skill Level: Any Skill Level

Familiarity with Kubernetes, Docker, and GitHub

Service accounts and namespaces allow you to limit pod and user permission in Kubernetes. Audit logs provide insight into what accounts are accessing what resources. Learn how to use these Kubernetes features!

Step-by-step

  1. Overview

    In this walkthrough, you create a namespace and add 2 service accounts, each with its own role, to it. One account can read secrets, Kubernetes objects that store sensitive information, and the other cannot. Each of these service accounts is attached to its own pod. Both of these pods run the same container image. At run time, secrets are read from within the container by using a curl command. The authorization to access the secret is determined by an API token that is mounted within the container. This token is generated by using the pod’s service account name. The API access is logged to a file, which contains the secret access by these service accounts.

    Secrets give the ability to securely store data outside of a pod definition. Service accounts give the ability to grant and revoke access to resources from pods. Since both secrets and service accounts are scoped to a namespace, you can access them by using only accounts with appropriate permissions to the namespace. Additionally, audit logs allow traceability of accounts that access resources over the API.

  2. Clone the Git Repo

    Clone the Github project: https://github.com/IBM/k8s-service-accounts
    The root directory of this project contains all the sample YAML files, the docker directory contains the Dockerfile used by the pod.

    Each YAML file is named based on the resource it creates. For example the api-reader-cluster-roles.yaml
    file defines the cluster roles that you use in this project.

    The api-reader-all-in-one.yaml file contains all the definitions in a single file. You can review the resources from a single location, however, in this guide you create each resource individually.

  3. Start Minikube

    For Kubernetes to honor the service accounts’ roles, you must enable Role-Based Access Control (RBAC) support in Minikube.

    Because the audit log configuration options are different in Kubernetes 1.6 and 1.7, confirm which Kubernetes version you use. Run the following command:

     

    $ kubectl version
    Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:40:50Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
    Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-05-09T23:22:45Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

     In the output, you can see that the server version is 1.6.0.

    If you use version 1.6.0, start Minikube by running this command:

    $ minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.Path=/var/log/apiserver/audit.log
    Starting local Kubernetes v1.6.0 cluster...
    Starting VM...
    SSH-ing files into VM...
    Setting up certs...
    Starting cluster components...
    Connecting to cluster...
    Setting up kubeconfig...
    Kubectl is now configured to use the cluster.

    If you use Kubernetes version 1.7, start Minikube by running this command (Not released on Minikube at the time of publication):

    $ minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.LogOptions.Path=/var/log/apiserver/audit.log
  4. Create the Namespace

    When you run Kubernetes as a multi-tenant or multi-project environment, you create namespaces to scope resources. You create items like pods, secrets, and service accounts within a namespace and can set resource quotas at namespace level. You can learn more about namespaces in the Kubernetes docs.

    Create a new namespace. Use the following resource definition:

    apiVersion: v1
    kind: Namespace
    metadata:
    name: dev

    You can find the namespace definition in the api-reader-dev-namespace.yaml file.

    To create the “dev” namespace, run this command:

    $ kubectl create -f api-reader-dev-namespace.yaml 
    namespace "dev" created

    To allow kubectl to run commands in the dev namespace, change the kubectl context. The following command obtains the current cluster name and updates the context that you use to run commands on it:

    $ kubectl config set-context $(kubectl config current-context) --namespace=dev
    Context "minikube" set.

    The current cluster name is “minikube.”

  5. Create the Secret

    A secret is an object that is used to store sensitive information, like passwords and authentication keys. In this example, a user name and password are stored for demonstration purposes. Secret data must be encoded in base64. You can learn more about secrets in the Kubernetes docs.

    The secret is defined in the api-reader-secret.yaml file. Its contents follow:

    ---
    # Secret with base64 encoded values
    apiVersion: v1
    kind: Secret
    metadata:
    name: api-access-secret
    type: Opaque
    data:
    username: YWRtaW4=
    password: cGFzc3dvcmQ=

    To create the secret, run this command:

    $ kubectl create -f api-reader-secret.yaml 
    secret "api-access-secret" created

    To verify that the secret was created, run this command:

    $ kubectl get secrets api-access-secret
    NAME TYPE DATA AGE
    api-access-secret Opaque 2 4h

     

  6. Create the Service Accounts

    You can attach service accounts to pods and use it to access the Kubernetes API. If a service account is not set in the pod definition, the pod uses the default service account for the namespace. Files that are named token, ca.crt, and namespace are automatically mounted in the /var/run/secrets/kubernetes.io/serviceaccount/ directory of each container. Their contents are based on the service account name that you provide. Kubernetes docs

    Note: The secrets that are shown in the /var/run/secrets/kubernetes.io/serviceaccount/ directory are service account specific secrets that are mounted by the Kubernetes system, not the secret that you created. The access to this secret does not indicate that the pod can access other secrets with this token.

    The service accounts are defined in api-reader-service-accounts.yaml file. Its contents follow:

    ---
    # Service account for preventing API access
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: no-access-sa
    ---
    # Service account for accessing secrets API
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: secret-access-sa

    To create these service accounts, run this command:

    $ kubectl create -f api-reader-service-accounts.yaml 
    serviceaccount "no-access-sa" created
    serviceaccount "secret-access-sa" created

    To confirm that the accounts exist, run this command:

    $ kubectl get serviceaccounts
    NAME SECRETS AGE
    default 1 24m
    no-access-sa 1 5m
    secret-access-sa 1 5m

     

  7. Create the Cluster Roles

    A cluster role defines a set of permissions that is used for accessing resources, such as pods and secrets. Cluster roles are scoped to the cluster. The cluster roles that are defined here are attached to the service accounts via a role binding in subsequent steps. Using a role binding instead of a cluster role binding scopes the permissions to a namespace. For more information about roles, see the Kubernetes docs.

    The cluster roles are defined in the api-reader-cluster-roles.yaml file. Its contents are shown:

    ---
    # A role with no access
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
    name: no-access-cr
    rules:
    - apiGroups: [""] # "" indicates the core API group
    resources: [""] verbs: [""]
    ---
    # A role for reading/listing secrets
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
    name: secret-access-cr
    rules:
    - apiGroups: [""] # "" indicates the core API group
    resources: ["secrets"] verbs: ["get", "list"]

    To create the cluster roles, run this command:

    $ kubectl create -f api-reader-cluster-roles.yaml
    clusterrole "no-access-cr" created
    clusterrole "secret-access-cr" created

    To verify that the roles were created, run this command:

    $ kubectl get clusterroles
    NAME KIND
    ...
    no-access-cr ClusterRole.v1beta1.rbac.authorization.k8s.io
    secret-access-cr ClusterRole.v1beta1.rbac.authorization.k8s.io
    system:basic-user ClusterRole.v1beta1.rbac.authorization.k8s.io
    ...

     

  8. Create the Role Bindings

    To apply cluster roles to service accounts, create role bindings that connect them. When you bind a c;ister role to a service account, the permissions that you defined in a role are granted to the account. Kubernetes docs

    The role bindings are defined in the api-reader-role-bindings.yaml file, as shown in the following text:

    ---
    # The role binding to combine the no-access service account and role
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
    name: no-access-rb
    subjects:
    - kind: ServiceAccount
    name: no-access-sa
    roleRef:
    kind: ClusterRole
    name: no-access-cr
    apiGroup: rbac.authorization.k8s.io

    ---
    # The role binding to combine the secret-access service account and role
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
    name: secret-access-rb
    subjects:
    - kind: ServiceAccount
    name: secret-access-sa
    roleRef:
    kind: ClusterRole
    name: secret-access-cr
    apiGroup: rbac.authorization.k8s.io

    To create these role bindings, run this command:

    $ kubectl create -f api-reader-role-bindings.yaml 
    rolebinding "no-access-rb" created
    rolebinding "secret-access-rb" created

    To view the role bindings, run this command:

    $ kubectl get rolebindings
    NAME KIND
    no-access-rb RoleBinding.v1beta1.rbac.authorization.k8s.io
    secret-access-rb RoleBinding.v1beta1.rbac.authorization.k8s.io

     

  9. Review the Dockerfile

    The Dockerfile for this example contains a single pod. When you run the Dockerfile, you install curl in the container and copy a runtime.sh script, which contains the commands that the container runs at startup, to it.

    To build this Docker image locally, switch to the Docker directory and run this command:

    docker build -t <docker_image_tag> .

    The Dockerfile contains the following code:

    #ibmcloudprivate/k8s-service-accounts
    FROM ubuntu

    # Copy files
    COPY runtime.sh /
    # Modify file permissions
    RUN chmod +x runtime.sh


    # Install curl
    RUN apt-get update
    RUN apt-get install curl -y

    # Run script on startup
    CMD [ "/runtime.sh" ]

    The runtime.sh script contains the following code:

    #!/bin/bash

    # Read mounted files
    KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token)
    NAMESPACE=$(</var/run/secrets/kubernetes.io/serviceaccount/namespace)

    # Resource to get in API [pods/secrets]RESOURCE="secrets"

    # If an argument was set
    if [ "$#" -ge 1 ]; then
    NAMESPACE="$1"
    fi

    #Curl against the resource
    echo curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/api/v1/namespaces/$NAMESPACE/$RESOURCE
    curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/api/v1/namespaces/$NAMESPACE/$RESOURCE
    echo ""

    # Don't wait on user exec
    if [ "$#" -lt 1 ]; then
    # Sleep forever so the pod doesn't crash
    while [ true ]; do
    sleep 10
    done
    fi

     

  10. Create the Pods

    You use service accounts to access the API from within a pod. In this model, your container accesses the secrets API at runtime instead of referencing secrets in the pod definition YAML. Secrets that are referenced in the YAML spec do not abide by the service account permissions.

    Service account information is automatically mounted to the /var/run/secrets/kubernetes.io/serviceaccount/ directory and consists of the ca.crt, namespace, and token files.

    The pods are defined in api-reader-pods.yaml. If you built the docker image locally in the Review the Dockerfile step, replace the ibmcloudprivate/k8s-service-accounts image parameter values in this YAML with the name of the image that you built.

    ---
    # Create a pod with the no-access service account
    kind: Pod
    apiVersion: v1
    metadata:
    name: no-access-pod
    spec:
    serviceAccountName: no-access-sa
    containers:
    - name: no-access-container
    image: ibmcloudprivate/k8s-service-accounts

    ---
    # Create a pod with the secret-access service account
    kind: Pod
    apiVersion: v1
    metadata:
    name: secret-access-pod
    spec:
    serviceAccountName: secret-access-sa
    containers:
    - name: secret-access-container
    image: ibmcloudprivate/k8s-service-accounts

    To create these pods, run this command:

    $ kubectl create -f api-reader-pods.yaml
    rolebinding "no-access-rb" created
    rolebinding "secret-access-rb" created

    To verify that the role bindings were created, run the following command:

    $ kubectl get pods
    NAME READY STATUS RESTARTS AGE
    no-access-pod 0/1 ContainerCreating 0 9s
    secret-access-pod 0/1 ContainerCreating 0 9s

     

  11. View the Pod Output

    This step verifies that one of the pods can access the secrets API and the other cannot.

    Verify that both of the pods that you created are in the running state by running this command:

    $ kubectl get po
    NAME READY STATUS RESTARTS AGE
    no-access-pod 1/1 Running 0 57s
    secret-access-pod 1/1 Running 0 57s

    Run this command to obtain the logs from the no-access pod:

    $ kubectl logs no-access-pod
    curl -sSk -H Authorization: Bearer <token> https://10.0.0.1:443/api/v1/namespaces/dev/secrets
    User "system:serviceaccount:dev:no-access-sa" cannot list secrets in the namespace "dev".

    The command output shows that access is denied.

    Run this command to view the logs from the secret-access pod. The command output includes the secret that you created.

    $ kubectl logs secret-access-pod
    curl -sSk -H Authorization: Bearer <token> https://10.0.0.1:443/api/v1/namespaces/dev/secrets
    {
    "kind": "SecretList",
    "apiVersion": "v1",
    "metadata": {
    "selfLink": "/api/v1/namespaces/dev/secrets",
    "resourceVersion": "2029"
    },
    "items": [
    {
    "metadata": {
    "name": "api-access-secret",
    "namespace": "dev",
    "selfLink": "/api/v1/namespaces/dev/secrets/api-access-secret",
    "uid": "4e965c40-550f-11e7-9f14-080027ab725e",
    "resourceVersion": "1917",
    "creationTimestamp": "2017-06-19T16:49:48Z"
    },
    "data": {
    "password": "cGFzc3dvcmQ=",
    "username": "YWRtaW4="
    },
    "type": "Opaque"
    },
    ...
    }

     

  12. Try to Access Secrets on Another Namespace

    You can use Kubernetes to run other commands on containers in your pod. When you run the runtime.sh script in the container, you can pass a namespace parameter to it. If you don’t pass a namespace parameter to it, it uses the namespace of the service account.

    Run the runtime.sh script and specify the default namespace, as shown in this command:

    $ kubectl exec -it secret-access-pod /runtime.sh default
    curl -sSk -H Authorization: Bearer <token> https://10.0.0.1:443/api/v1/namespaces/default/secrets
    User "system:serviceaccount:dev:secret-access-sa" cannot list secrets in the namespace "default".

    Because the service account is scoped to the dev namespace, it is not authorized to use the default namespace. Its access to the default namespace is denied.

     

  13. View the Audit Logs

    Audit logs allow administrators to view the particular resources that an account has accessed. The audit logs are stored on the system that hosts Minikube. You set the log storage location when you started Minikube in by using the ` –extra-config` parameter.

    Run this command to view the logs:

    $ minikube ssh sudo cat /var/log/apiserver/audit.log | grep "system:serviceaccounts:dev"
    2017-06-20T11:29:23.510487235Z AUDIT: id="7a09d503-cf8c-48f8-8b4a-d92f775d6dc5" ip="172.17.0.2" method="GET" user="system:serviceaccount:dev:no-access-sa" groups="\"system:serviceaccounts\",\"system:serviceaccounts:dev\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="dev" uri="/api/v1/namespaces/dev/secrets"
    2017-06-20T11:29:24.550061517Z AUDIT: id="f1ac0d89-a646-4fd4-aeb6-d90be0c2c900" ip="172.17.0.3" method="GET" user="system:serviceaccount:dev:secret-access-sa" groups="\"system:serviceaccounts\",\"system:serviceaccounts:dev\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="dev" uri="/api/v1/namespaces/dev/secrets"
    2017-06-20T11:39:22.314592214Z AUDIT: id="3d5253ba-b1a8-4da4-baf2-7f81b85c5905" ip="172.17.0.2" method="GET" user="system:serviceaccount:dev:secret-access-sa" groups="\"system:serviceaccounts\",\"system:serviceaccounts:dev\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/secrets"

    The log shows the URI of the accessed resource and the user who accessed it. For example, the last item in the log shows that the service account in the dev namespace that is named secret-access-sa (user=”system:serviceaccount:dev:secret-access-sa“) made a request for the secrets in the default namespace (uri=”/api/v1/namespaces/default/secrets“)

     

  14. Conclusion

    Service accounts are a powerful tool for cluster administration because you can use them to control and view access of resources in Kubernetes. You can use them to limit access to a particular namespace. You should limit pods to access only to what they need. The ability to know who accessed which resources and when they accessed them provides insight into cluster activity.

Join The Discussion