Archived content

Archive date: 2024-06-02

This content is no longer being updated or maintained. The content is provided “as is.” Given the rapid evolution of technology, some content, steps, or illustrations may have changed.

The KCP (Kubernetes Control Plane) project is an open source project that provides a highly available, scalable, and reliable control plane for Kubernetes clusters. The project is designed to address various issues and challenges in managing and scaling large Kubernetes clusters and provide a stable and efficient control plane. In simpler words, KCP is a minimal Kubernetes API server that can also be categorized as follows:

A control plane for many independent, isolated “clusters,” known as workspaces
Enables API service providers to offer APIs centrally by using multi-tenant operators
Easy API consumption for users in their workspaces
Flexible scheduling of workloads to create physical clusters

Hub-and-spoke architecture is a popular design pattern used in distributed systems to facilitate communication between different components of the system. In this architecture, a central hub acts as a communication node, connecting multiple peripheral components, or spokes, to each other. KCP’s capabilities can be used to facilitate communication between the geo-distributed clusters, where a Kubernetes cluster designated as a hub can be used as a communication node to push down custom resource objects to the connected spoke sites (clusters). To enable this behavior in the distributed clusters, we need to understand KCP concepts like workspace, namespace, SyncTarget, location, and placement policies.

Workspace: Workspaces are used to enable multi-tenancy in a system. These workspaces are HTTPS endpoints similar to Kubernetes clusters, which can be accessed by standard Kubernetes client tools such as client-go, controller-runtime, and user interfaces like kubectl, helm, and web console.
SyncTarget: A sync target is a representation of Kubernetes clusters that are attached to a KCP installation to execute workload objects from the users' workspaces. The task of synchronizing data between KCP and a particular cluster is carried out by a syncer, which is installed on a SyncTarget object.
Location: Location refers to a group of SyncTarget objects that are chosen based on instance labels. It provides users with labels that may be different from the instance labels, and these labels can be used to identify, describe, and select specific locations for placing user namespaces onto SyncTarget.
Placement policies: A placement defines a rule for selecting a single location using location labels and binding that selected location to multiple namespaces within a user workspace. In cases where a workspace has multiple namespaces, users can create multiple placements to assign particular namespaces to specific locations.
Namespace: The KCP namespace maps to an actual namespace in the physical cluster. The resources created in it are synchronized to the corresponding namespace at the physical cluster.

The following architecture diagram depicts a sample hub-and-spoke design of the clusters in the KCP context.

Prerequisites

To follow along, you need to have:

Kubectl, a command-line tool that allows you to run commands against Kubernetes clusters.
Kind, a tool for running local Kubernetes clusters using a Docker container.
Visit KCP’s latest release page and download KCP and the kubectl-kcp-plugin that match your operating system and architecture (this tutorial is tested on v0.11.0).

Steps

Open the terminal and create three Kubernetes clusters using Kind.

 kind create cluster -n hub

Output for command "kind create cluster -n hub"

 kind create cluster -n spoke-1

Output for command "kind create cluster -n spoke-1"

 kind create cluster -n spoke-2

Output for command "kind create cluster -n spoke-2"

Verify that the clusters were created.
```
 kind get clusters
```
Execute the following command to run the KCP.
```
 kcp start
```
Open a new tab in the terminal, and export the KUBECONFIG to the KCP’s config. During its startup, KCP generates a kubeconfig in the .kcp/admin.kubeconfig file.
```
 export KUBECONFIG=/<path_to_kubeconfig> /.kcp/admin.kubeconfig
```
Create a workspace named “hubnspoke” under which our clusters specific custom resources will be created. A kubeconfig file can contain multiple contexts, each with its own set of configuration details. This allows users to easily switch between different workspaces without having to enter the connection details each time.
```
 kubectl workspace create hubnspoke --type universal --enter
```
```
 kubectl workspace create-context
```
You should be able to see the workspace with the help of the following command:
```
 kubectl ws tree
```
Create a manifest file for the syncer for each cluster and deploy it on each of the sites.

In the next step, we are going to create a manifest file for the syncer for each cluster and deploy it on each of the sites. A syncer is installed on a SyncTarget and is responsible for synchronizing data between KCP and the cluster. Thereafter, you should be able to use YAML generated manifests files for each of the sites.
```
 kubectl kcp workload sync hub --syncer-image ghcr.io/kcp-dev/kcp/syncer:v0.11.0 -o hub.yaml
```
```
 kubectl kcp workload sync spoke-1 --syncer-image ghcr.io/kcp-dev/kcp/syncer:v0.11.0 -o spoke-1.yaml
```
```
 kubectl kcp workload sync spoke-2 --syncer-image ghcr.io/kcp-dev/kcp/syncer:v0.11.0 -o spoke-2.yaml
```
To deploy the syncer pod on the cluster, export the KUBECONFIG to the default and then switch the cluster’s context to deploy the syncer pods individually.

Switch context to the hub cluster and deploy the syncer pod to the hub cluster.
```
 KUBECONFIG=~/.kube/config kubectl config use-context kind-hub
```
```
 KUBECONFIG=~/.kube/config kubectl apply -f hub.yaml
```
Switch context to the spoke-1 cluster and deploy the syncer pod to the spoke-1 cluster.
```
 KUBECONFIG=~/.kube/config kubectl config use-context kind-spoke-1
```
```
 KUBECONFIG=~/.kube/config kubectl apply -f spoke-1.yaml
```
Switch context to the spoke-2 cluster and deploy the syncer pod to the spoke-2 cluster.
```
 KUBECONFIG=~/.kube/config kubectl config use-context kind-spoke-2
```
```
 KUBECONFIG=~/.kube/config kubectl apply -f spoke-2.yaml
```

Check the newly created SyncTargets and then label the SyncTargets to refer them in the placement policies.

 kubectl get synctarget

Output for kubectl get synctarget

 kubectl label synctarget/hub name=hub –overwrite

Output for kubectl label synctarget/hub name=hub –overwrite

 kubectl label synctarget/spoke-1 name=spoke-1 –overwrite

Output for kubectl label synctarget/spoke-1 name=spoke-1 –overwrite

 kubectl label synctarget/spoke-2 name=spoke-2 –overwrite

Output for kubectl label synctarget/spoke-2 name=spoke-2 –overwrite

Create manifest files to create namespaces for the hub, spoke-1, and spoke-2 sites.

Write a manifest file for hub and then create it.

vi nsHub.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: hub-operator-system
  labels:
   name: hub

kubectl create -f nsHub.yaml

Output for creating a manifest file for hub

Write a manifest file for spoke-1 and then create it.

vi nsSpoke-11.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: spoke-1
  labels:
   name: spoke-1

kubectl create -f nsSpoke1.yaml

Output for creating a manifest file for spoke-1

Write a manifest file for spoke-2 and then create it.

vi nsSpoke-2.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: spoke-2
  labels:
   name: spoke-2

kubectl create -f nsSpoke2.yaml

Output for creating a manifest file for spoke-2

Verify that the namespaces were created.

kubectl get ns

Output for verifying that the namespaces were created

Create a location for each site with its instance selector matching the label of the respective SyncTarget.

Write a location manifest file for “hub” and create it.

vi locHub.yaml
apiVersion: scheduling.kcp.io/v1alpha1
kind: Location
metadata:
  name: location-hub
  labels:
  name: location-hub
spec:
instanceSelector:
 matchLabels:
  name: hub
resource:
  group: workload.kcp.io
  resource: synctargets
  version: v1alpha1

kubectl create -f locHub1.yaml

Output for creating a location manifest file for “hub”

Write a location manifest file for “spoke 1” and create it.

vi locSpoke1.yaml
apiVersion: scheduling.kcp.io/v1alpha1
kind: Location
metadata:
 name: location-spoke-1
  labels:
  name: location-spoke-1
spec:
instanceSelector:
   matchLabels:
   name: spoke-1
  resource:
    group: workload.kcp.io
    resource: synctargets
    version: v1alpha1

kubectl create -f locSpoke1.yaml

Output for creating a location manifest file for “spoke-1”

Write a location manifest file for “spoke 2” and create it.

vi locSpoke2.yaml
apiVersion: scheduling.kcp.io/v1alpha1
kind: Location
metadata:
 name: location-spoke-2
  labels:
  name: location-spoke-2
spec:
instanceSelector:
  matchLabels:
  name: spoke-2
resource:
 group: workload.kcp.io
 resource: synctargets
 version: v1alpha1

 kubectl create -f locSpoke2.yaml

Output for creating a location manifest file for “spoke-2”

Verify that the locations were created.

kubectl get locations

Output for verifying that the locations were created

Delete the default location.

kubectl delete location default

Create placement policies to bind the KCP namespace with the location.

Create the placement manifest file for “hub,” mapping the location label to the namespace label of the hub.

vi pp-n2l-hub.yaml
apiVersion: scheduling.kcp.io/v1alpha1
kind: Placement
metadata:
name: pp-hub
spec:
  locationResource:
   group: workload.kcp.io
   resource: synctargets
  version: v1alpha1
 locationSelectors:
 - matchLabels:
  name: location-hub
 namespaceSelector:
  matchLabels:
   name: hub
 locationWorkspace: root:hubnspoke

kubectl create -f pp-n2l-hub.yaml

Output for creating placement manifest file for “hub"

Create the placement manifest file for “spoke 1,” mapping the location label to the namespace label of the spoke 1.

vi pp-n2l-spoke1.yaml
apiVersion: scheduling.kcp.io/v1alpha1
kind: Placement
metadata:
  name: pp-spoke-1
spec:
 locationResource:
  group: workload.kcp.io
  resource: synctargets
  version: v1alpha1
locationSelectors:
  - matchLabels:
  name: location-spoke-1
 namespaceSelector:
   matchLabels:
  name: spoke-1
locationWorkspace: root:hubnspoke

kubectl create -f pp-n2l-spoke1.yaml

Output for creating placement manifest file for “spoke 1"

Create the placement manifest for “spoke 2,” mapping the location label to the namespace label of the spoke 2.

vi pp-n2l-spoke2.yaml
apiVersion: scheduling.kcp.io/v1alpha1
kind: Placement
metadata:
 name: pp-spoke-2
spec:
locationResource:
  group: workload.kcp.io
  resource: synctargets
  version: v1alpha1
locationSelectors:
- matchLabels:
  name: location-spoke-2
namespaceSelector:
 matchLabels:
  name: spoke-2
locationWorkspace: root:hubnspoke

kubectl create -f pp-n2l-spoke2.yaml

Output for creating placement manifest file for “spoke 2"

Get the placements

kubectl get placements

Output to get the placements"

Finally, run the following command (continuing in the hubnspoke workspace) to create a binding to the workload APIs export. We just need the API Binding and not placement which is automatically created by the command. Thereafter, we can delete the placement.
```
kubectl kcp bind compute hubnspoke
```

Create a deployment to a particular KCP namespace in the “hubnspoke” workspace and verify that it gets synced to your cluster:

pankajthorat@Pankajs-MacBook-Pro blog % kubectl create deployment --image=gcr.io/kuar-demo/kuard-amd64:blue --port=8080 kuard -n spoke-1
deployment.apps/kuard created

pankajthorat@Pankajs-MacBook-Pro blog % kubectl create deployment --image=gcr.io/kuar-demo/kuard-amd64:blue --port=8080 kuard -n spoke-2
deployment.apps/kuard created

Output for verifying and creating deployment to a particular KCP namespace in the “hubnspoke"

Verify individual deployments and their results using the following commands:

kubectl config use-context kind-spoke-2

kubectl get deployment -A | grep kuard

Output for verifying deployments and their results

kubectl config use-context kind-spoke-1

kubectl get deployment -A | grep kuard

Output for verifying deployments and their results

Summary

In this tutorial, we have successfully deployed a one-to-one hub-and-spoke deployment model using Kubernetes Control Plane (KCP). This deployment model allows the Kubernetes client, which is integrated into the hub operator, to select a specific cluster for workload deployment using a namespace that is mapped to that physical cluster. By using the Kubernetes client to select the appropriate cluster based on the namespace mapping, we can ensure that workloads are automatically deployed to their intended locations without the need for manual intervention. This approach allows us to take into account important factors such as workload and available resource requirements, ensuring that workloads are deployed efficiently and effectively across the KCP clusters. Overall, the successful implementation of the one-to-one deployment model within the KCP has proven to be a significant step forward in our efforts to improve workload deployment and management. This approach can be refined and optimized to further enhance its capabilities and benefits for specific use cases.

Products

Languages

Technologies

Events

External Resources

Enabling cross-cluster connectivity through hub-and-spoke design with KCP configurations

Prerequisites

Steps

Summary