Kubernetes with OpenShift World Tour: Get hands-on experience and build applications fast! Find a workshop!

Configuring Kubernetes clusters for multiple tenants, part 1

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It is very popular with developers as a container orchestration platform for managing containerized workloads and services. Usually, you can provision a local Kubernetes cluster or create it through the public cloud provider and launch a workload on the cluster. For example, you can easily get a Kubernetes cluster from the IBM Cloud Kubernetes Service.

In some cases, a stand-alone Kubernetes cluster doesn’t meet your organization’s isolation requirements. For example, if you need to engage different internal teams and different products on the cloud, you need a type of isolation to avoid any noise, conflict, and security issues across different business units. Launching multiple clusters is a good choice to provide a strict isolation to different teams, but the cost is very high. Also, it is hard to share some common internal services across multiple Kubernetes clusters. Another option is to share one Kubernetes cluster. The good news is that Kubernetes already has several isolation features that support multiple tenants on a single Kubernetes cluster.

This tutorial introduces how to configure a Kubernetes cluster to support multiple tenants. Examples are based on the IBM Cloud Kubernetes service.

Before getting into the tutorial, start by understanding the term tenant in Kubernetes.

A tenant represents a group of Kubernetes users. It has certain permissions to access a group of resources in a Kubernetes cluster, but cannot access other resources without necessary permissions. Resources in a Kubernetes cluster are isolated by tenants. Tenants can allocate a certain quota to consume resources in a Kubernetes cluster. Also, an administrator for a tenant can manage permissions of a group of resources for the other tenant users.

You can leverage several Kubernetes features to meet your requirements for multiple tenants. You can use Kubernetes namespaces to group other Kubernetes resources (for example, pods, deployment, services). You can use role-based access control (RBAC) to define a particular role for a Kubernetes resource, and bind the role to target users to grant their permission of related resources. Also, you can define a network policy to control inbound and outbound traffic for a namespace. Finally, you can configure a Kubernetes resource quota to control the use for each namespace.

Prerequisites

To prepare for the steps in this tutorial, you need to complete the following tasks:

Estimated time

Completing all the steps in this tutorial takes about one hour.

Steps to set up multiple tenants on a Kubernetes cluster

The following steps use an IBM Cloud Kubernetes cluster as example to show how to configure a cluster for multiple tenants.

  1. Provision one IBM Cloud Kubernetes cluster called multiple-tenants-cluster through the IBM Cloud Console. You can refer to steps in Getting started with IBM Cloud Kubernetes Service. Assume that sam@company.com is the user who create the cluster. Sam is the administrator for the whole cluster and can do anything on the cluster. You can get the detailed permissions from Kubernetes resource permissions per RBAC role (admin and cluster-admin column).

  2. When the cluster is created, the IBM Cloud Kubernetes Service automatically creates a cluster role binding ibm-admin, which assigns the creator sam@company.com with a particular cluster-role as cluster-admin. Therefore, sam@company.com acts as the cluster admin and can manage tenants in this cluster. Use the following commands to find the ClusterRole definition of cluster-admin:

    >> kubectl get clusterrole cluster-admin -o yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      ......
      name: cluster-admin
    rules:
    - apiGroups:
      - '*'
      resources:
      - '*'
      verbs:
      - '*'
    - nonResourceURLs:
      - '*'
      verbs:
      - '*'
    
  3. List the current cluster admin authorities through the ClusterRoleBinding definition:

     >> kubectl get clusterrolebinding ibm-admin -o yaml
     apiVersion: rbac.authorization.k8s.io/v1
     kind: ClusterRoleBinding
     metadata:
     ......
     name: admin
     roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: cluster-admin
     subjects:
     - apiGroup: rbac.authorization.k8s.io
     kind: User
     name: IAM#sam@company.com
    
  4. The cluster admin sam@company.com can create a namespace called group-1 with the command kubectl create namespace group-1. The following section treats the namespace group-1 as a Kubernetes resource group for a tenant.

  5. (Optional) The cluster admin sam@company.com can assign a reasonable resource quota to different tenants to prevent resource shortage.

    See the example of group-1 by running the following command. Of course, you can configure the number based on your requirements and the worker node size of the cluster:

     cat << EOF | kubectl apply -f -
     ---
     apiVersion: v1
     kind: ResourceQuota
     metadata:
       name: compute-resources
       namespace: group-1
     spec:
       hard:
         pods: "2"
         requests.cpu: "1"
         requests.memory: 1Gi
         limits.cpu: "2"
         limits.memory: 2Gi
         requests.nvidia.com/gpu: 4
     EOF
    
  6. In this step and the next, assume that the cluster admin sam@company.com wants to add another user named jan@company.com into the cluster as a tenant admin to manage namespace called group-1.

    First, sam@company.com needs to onboard jan@company.com to the IBM Cloud Kubernetes service so that jan@company.com can have the right permissions to download the cluster kubeconfig file before running the kubectl command. Follow the Assigning IBM Cloud IAM roles with the console to add the Platform access -- viewer permission.

    You can use the IBM Cloud user interface or run the following command:

     ibmcloud account user-invite jan@company.com
     ibmcloud iam user-policy-create jan@company.com --roles Viewer --service-name containers-kubernetes
    

    Then, jan@company.com can download the cluster configuraton with ibmcloud cs cluster config --cluster <cluster-name>.

  7. Now, the cluster admin sam@company.com can grant the tenant admin authority to jan@company.com by binding the user ID to the admin role on a selected namespace:

     cat << EOF | kubectl apply -f -
     ---
     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: tenant-admin
       namespace: group-1
     subjects:
     - kind: User
       name: IAM#jan@company.com
       apiGroup: rbac.authorization.k8s.io
     roleRef:
       kind: ClusterRole
       name: admin
       apiGroup: rbac.authorization.k8s.io
     EOF
    

    If you want to get detailed settings for admin, you can check its role definition by running kubectl get clusterrole admin -o yaml.

    Up until now, jan@company.com was the admin of the group-1 namespace but couldn’t access any other namespaces.

  8. In this step, the cluster admin sam@company.com wants to add another user tom@company.com into the cluster as a tenant user of the namespace group-1.

    Similar to the previous steps, sam@company.com needs to grant IAM Platform access -- viewer permission to tom@company.com.

    In the IBM Cloud Kubernetes Service, there already edit roles defined that you can use for a tenant user who deploys the application inside a namespace. If you are interested in detailed setting for edit, check its role definition with the kubectl get clusterrole edit -o yaml command.

    After you granted a tenant administrator role to jan@company.com for the group-1 namespace in the previous steps, jan@company.com has the ability to log into the cluster to grant tom@company.com as a tenant user of the group-1 namespace :

     cat << EOF | kubectl apply -f -
     ---
     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: tenant-user
       namespace: group-1
     subjects:
     - kind: User
       name: IAM#tom@company.com
       apiGroup: rbac.authorization.k8s.io
     roleRef:
       kind: ClusterRole
       name: edit
       apiGroup: rbac.authorization.k8s.io
     EOF
    

    Now, tom@company.com can log in and deploy an application within namespace group-1 without the access to any other namespaces.

Steps to configure the network for multiple tenants

For a cluster with multiple tenants, tenant administrators might want to limit the service inside their own namespaces only, denying access from any other namespaces.

You could implement this requirement with Kubernetes network policies. See Network Policies.

In the following example, Jan, a tenant admin, configures the network policies on her namespace to deny access from other namespaces except kube-system which is a system namespace that hosts a lot system pods in IBM Cloud Kubernetes service. In this situation, you need to allow the access of kube-system to make sure you can expose your application’s external access through Ingress. (Of course a tenant admin like Jan can also follow the steps in Network Policies to define more network policies on her group-1 namespace.)

  1. Label kube-system with namespace: system:

    >> kubectl edit namespace kube-sytem
    apiVersion: v1
    kind: Namespace
    metadata:
    ......
    labels:
      namespace: system
     name: kube-system
    spec:
    finalizers:
    - kubernetes
    status:
    phase: Active
    
  2. Create a network policy on namespace group-1 that allows access from only the namespaces that are labeled with namespace:system:

     cat << EOF | kubectl apply -f -
     ---
     kind: NetworkPolicy
     apiVersion: networking.k8s.io/v1
     metadata:
       namespace: group-1
       name: deny-from-other-namespaces
     spec:
       podSelector: {}
       ingress:
       - from:
         - podSelector: {}
         - namespaceSelector:
             matchLabels:
               namespace: system
     EOF
    

Summary

From following the steps in this tutorial, you can see that Kubernetes provides some helpful features for implementing multiple tenants on a single Kubernetes cluster, for example, namespace, role-based access control, resource quota, and network policy. With these options, you can achieve the right level of isolation and resource sharing across multiple tenants. In addition, the Kubernetes community is working to develop more capabilities to support multiple tenants. You can follow the Kubernetes Multi-tenancy Special Interest Group community at kubernetes-sigs/multi-tenancy community to get updates.

Yu Zhuang
Ying Liu