Originally published as IBM Developer Recipe here on 10 February 2021 by Hendrik van Run, Hugh Hockett and Christopher Liebl.
Overview
IBM Cloud Pak System 2.3.3.3 includes a Tech Preview that allows for the deployment of OpenShift clusters across multiple IBM Cloud Pak Systems. This tutorial describes the topologies available and includes step-by-step deployment instructions.
This article assumes you are familiar with the following aspects:
- IBM Cloud Pak System
- IBM Cloud Pak System accelerators
- Red Hat OpenShift Container Platform 4.6
Furthermore, you must have two or more IBM Cloud Pak Systems at your disposal. And finally, you must meet the minimum versions specified below.
- IBM Cloud Pak System 2.3.3.3
- OpenShift Container Platform accelerator version 4.6.0.0
Introduction
IBM Cloud Pak System comes with built-in support for automated provisioning and configuring of Red Hat OpenShift Container Platform, making it the perfect platform for on-premises deployment of IBM Cloud Paks and Red Hat OpenShift clusters.
In IBM Cloud Pak System 2.3.3.0 and earlier, a deployed OpenShift cluster could not span multiple IBM Cloud Pak Systems. IBM Cloud Pak System 2.3.3.3 includes a Tech Preview that allows for the deployment of OpenShift clusters across multiple IBM Cloud Pak Systems. This enables clients to deploy a stretched OpenShift cluster across multiple systems and even multiple data centers, dramatically improving the quality of service of the cluster.
Prerequisites
This article assumes you are familiar with the following aspects:
- IBM Cloud Pak System
- IBM Cloud Pak System accelerators
- Red Hat OpenShift Container Platform 4.6
As highlighted in the Introduction section, IBM Cloud Pak System must be at version 2.3.3.3 or higher. In addition, you must have the OpenShift Container Platform accelerator version 4.6.0.0 available from the catalog of the system.
Deploying a stretched OpenShift cluster also has requirements including no more than 10ms of latency between nodes for etcd to function properly. You can learn more about OpenShift cluster topologies and requirements from the blog post Disaster Recovery Strategies for Applications Running on OpenShift.
Enabling the Tech Preview feature
Note that the support for OpenShift 4 stretched clusters is not enabled by default in IBM Cloud Pak System 2.3.3.3. To enable this Tech Preview feature, run the following command on each of the IBM Cloud Pak Systems you intend to use. Make sure to pass in credentials of a user (userid:password) that has the Workload resources administration : Manage workload resources (Full permission) role enabled.
# curl -k -u userid:password -L -H 'Content-Type: application/json' -d \
'{"development_view": false,"enableStretchHelper": true}' \
-X PUT 'https://<PSM_IP>/cps/api/v1/enableDevelopmentCPSUI'
{"development_view":false,"enableStretchHelper":true}
#
Stretched OpenShift cluster topologies available
With IBM Cloud Pak System, an OpenShift cluster can span three IBM Cloud Pak Systems, or two IBM Cloud Pak Systems and another system. Each IBM Cloud Pak System system can support one master node and one or more worker nodes. The other system can only support one master node.
In either configuration, one of the IBM Cloud Pak System systems is referred to as the primary system. The other IBM Cloud Pak Systems are referred to as the remote systems. Each IBM Cloud Pak System manages its own resources (compute, storage, networking, etc). That is, there is no direct visibility on the IBM Cloud Pak System console to the master node or worker nodes on a remote system.
The relationship between the PrimaryHelper and RemoteHelpers in a stretched cluster deployment is similar to the relationship between a PrimaryHelper and SecondaryHelper in a single system deployment. That is, the configuration files are kept in sync on all helper nodes such that any helper node can serve as the access point to the cluster.
Note:
- In a single IBM Cloud Pak System deployment with two helper nodes, high availability between the PrimaryHelper and the SecondaryHelper is automatically supported using keepalived and a floating IP address. If the PrimaryHelper fails, control is automatically transferred to the SecondaryHelper. As such, access to the OpenShift cluster is not impacted by a helper node failure.
- In a stretched cluster deployment, helper HA is only supported between the PrimaryHelper and the first RemoteHelper. In addition, keepalived requires that the helper nodes be on the same subnet.
Stretched OpenShift cluster across three IBM Cloud Pak Systems
To support high availability of an OpenShift cluster, the master nodes (control plane) must be distributed across three availability zones. Figure 1 shows an OpenShift cluster that spans three IBM Cloud Pak systems. Each IBM Cloud Pak System resides in its own data center, satisfying the requirement for three availability zones.
Figure 1: Stretched OpenShift cluster across three IBM Cloud Pak Systems
Stretched OpenShift cluster across two IBM Cloud Pak Systems and IBM Cloud
Many clients only have two data centers, so how can you deploy a stretched clusters in such a scenario? Here, a public cloud provider like IBM Cloud can represent the third availability zone. The OpenShift stretched cluster can have one of the three master nodes deployed there. Of course, the third master virtual machine could also be deployed elsewhere as long as it truely represents a third availability zone.
Figure 2: Stretched OpenShift cluster across two IBM Cloud Pak Systems and IBM Cloud
Stretched OpenShift cluster across two IBM Cloud Pak Systems
As explained previously, a stretched OpenShift cluster should span three availability zones. With just two IBM Cloud Pak Systems in two data centers, we cannot meet that requirement. As shown in Figure 3, you would always end up with two masters in one of the two availability zones. So if that zone would suffer a (temporary) outage, the control plane of the OpenShift cluster would be impacted.
However using just two IBM Cloud Pak Systems does represent a valid scenario for test and evaluation purposes. Which is why we opted to describe the step-by-step process to deploy this topology in this tutorial.
Figure 3: Stretched OpenShift cluster across two IBM Cloud Pak Systems
Deploying a stretched OpenShift cluster across two IBM Cloud Pak Systems
Deploying a stretched OpenShift cluster across two IBM Cloud Pak Systems
Deploying OpenShift Container Cluster accelerator on the Remote IBM Cloud Pak System
-
Go to the Remote IBM Cloud Pak system. On the Welcome page, select Provision accelerators.
Figure 4: IBM Cloud Pak System Welcome Screen
-
On the Provision accelerators page, select the OpenShift Container Platform accelerator.
Figure 5: Select the OpenShift Container Platform accelerator
-
Under Deployment options, select Customized and click Continue.
Figure 6: Opt for a "Customized" deployment of the OpenShift Container Platform accelerator
-
On the Configure deployment page, enter a deployment name and helper node credentials. In the Red Hat OpenShift Container Platform cluster installation type, select Red Hat OpenShift Container Platform stretched cluster remote. For the Number of master nodes, enter 1 and click Configure nodes.
Figure 7: Configure the deployment of the remote OpenShift Container Platform accelerator
-
On the Configure nodes tab, select the number of worker nodes (1 in our case) and click OpenShift options. As a result of a minor limitation in the Tech Preview, this action takes you straight to Review and Deploy. You can ignore this limitation as it does not impact the deployment.
Figure 8: Configure the number of worker nodes of the remote OpenShift Container Platform accelerator
-
On the Review and deploy page, click Deploy. This displays the Environment is now deploying popup.
Figure 9: Deploy the remote OpenShift Container Platform accelerator
-
Select Manage accelerator instances.
Figure 10: The remote OpenShift Container Platform accelerator is now deploying
-
Monitor the progress of the deployment and wait for the Waiting for PrimaryHelper to initialize message in the History. It indicates that the remote OpenShift Container Platform accelerator deployment has completed and you can proceeed to the next step.
Figure 11: The remote OpenShift Container Platform accelerator deployment has completed
-
Switch to the Nodes page. Here, only the RemoteHelper node is in Running state as the master and worker nodes are in Stopped state. When the deployment on the Primary IBM Cloud Pak System gets launched, these nodes start automatically.
Figure 12: Only the RemoteHelper node of the OpenShift Container Platform accelerator deployment is Running
Note: If you were deploying a stretched OpenShift cluster across three IBM Cloud Pak Systems, you would have to repeat the previous steps 1-9 on the third system (which serves as another "Remote IBM Cloud Pak System" deployment).
Deploying OpenShift Container Cluster accelerator on the Primary IBM Cloud Pak System
-
Logon to the Primary Cloud Pak System. On the Welcome page, select Provision accelerators again.
Figure 13: IBM Cloud Pak System Welcome Screen
-
On the Provision accelerators page, select the OpenShift Container Platform accelerator.
Figure 14: Select the OpenShift Container Platform accelerator
-
Under Deployment options, select Customized and click Continue.
Figure 15: Opt for a "Customized" deployment of the OpenShift Container Platform accelerator
-
On the Configure deployment page, enter a deployment name and helper node credentials. In the Red Hat OpenShift Container Platform cluster installation type, select Red Hat OpenShift Container Platform stretched cluster primary. For the Number of master nodes, enter 2, thereby bringing the total number of master nodes across the Primary and Remote IBM Cloud Pak System to 3. Then click Configure nodes.
Figure 16: Configure the deployment of the Primary OpenShift Container Platform accelerator
-
On the Configure nodes tab, select the number of worker nodes (1 in our case). Now, click OpenShift options.
Figure 17: Configure the number of worker nodes of the Primary OpenShift Container Platform accelerator
-
On the OpenShift options page, enter the IP address and virtuser credentials of the RemoteHelper node that is running on the Remote IBM Cloud Pak System. Remember that you provided the virtuser credentials in Step 4 of the "Deploying OpenShift Container Cluster accelerator on the Remote IBM Cloud Pak System" section. Also note that you obtained the IP address at Step 9 (see Figure 12). Click Review and deploy to continue.
Figure 18: Deploy the Primary OpenShift Container Platform accelerator
-
On the Review and deploy page, click Deploy. It displays the Environment is now deploying popup.
Figure 19: Deploy the Primary OpenShift Container Platform accelerator
-
Click Manage accelerator instances in the popup.
Figure 20: The Primary OpenShift Container Platform accelerator is now deploying
-
Monitor the progress of the deployment and wait for the deployment to complete. In particular, wait for the Openshift post config completed message in the History. It indicates that the Primary OpenShift Container Platform accelerator deployment is complete and you can proceed to the next step.
Figure 21: The OpenShift Container Platform cluster has been deployed
-
Go to the Retrieve Kuberadmin password section in the bottom left-hand corner. Enter the root password of the PrimaryHelper node that you provided in Step 4 and click Submit.
Figure 22: Retrieving the kubeadmin password of the cluster
-
Copy the retrieved kubeadmin password to the clipboard.
Figure 23: Copy the kubeadmin password of the cluster
-
Select the Consoles from the drop down menu, then select OpenShift Container Platform web console. This launches the OpenShift Container Platform web console.
Figure 24: Launch the OpenShift Container Platform web console
-
Enter the kubeadmin credentials that you retrieved earlier to logon to the OpenShift web console.
Figure 25: Enter the kubeadmin password to logon to the OpenShift web console
-
In the navigation pane, expand Compute and select Nodes to display all the nodes of your OpenShift cluster. Confirm that the list of nodes matches the master and worker nodes depoyed on the Primary and Remote Cloud Pak Systems.
Figure 26: Review the nodes of your OpenShift cluster