Overview

Skill Level: Intermediate

IBM Cloud Pak System accelerates your implementation of on-premises Kubernetes platforms. It comes with support for automated deployment and configuration of Red Hat OpenShift Container Platform (OCP). This makes it the perfect platform for on-premises deployment of IBM Cloud Paks and Red Hat OpenShift clusters.

This tutorial walks you through steps for deployment of Red Hat OpenShift Container Platform 4.4 on IBM Cloud Pak System V2.3.3.0.

Ingredients

Overview

IBM Cloud Pak System accelerates your implementation of on-premises Kubernetes platforms. It comes with support for automated deployment and configuration of Red Hat OpenShift Container Platform (OCP). This makes it the perfect platform for on-premises deployment of IBM Cloud Paks and Red Hat OpenShift clusters!

This tutorial focuses on the deployment of Red Hat OpenShift Container Platform 4.4. For details on version 4.3, refer to the IBM Developer article Deploying Red Hat OpenShift 4.3 on IBM Cloud Pak System.

For Red Hat OpenShift, it is important to know that there are several different offerings available:

  • OpenShift Online

A fully managed public cloud offering for quickly deploying applications.

  • OpenShift Hosted Services

OpenShift clusters hosted on IBM Cloud, Amazon Web Services (AWS), and Azure.

  • OpenShift Container Platform (OCP)

An enterprise OpenShift cluster deployed on your own on-premises infrastructure (OpenShift Container Platform was previously called OpenShift Enterprise, but the name was changed with the release of version 3.3.).


A more detailed comparison of these offerings can be found on the OpenShift website. As IBM Cloud Pak System is an on-premises appliance, it only provides support for the OpenShift Container Platform offering. In this tutorial, you will learn how to deploy OpenShift Container Platform on IBM Cloud Pak System. Steps are written assuming that the IBM Cloud Pak System is at 2.3.3.0 firmware, and does not have direct access to the internet.

 
Prerequisites


Before you deploy your first OpenShift 4.4 cluster on IBM Cloud Pak System, few prerequisites need to be in place. For a good starting point on these prerequisites, see IBM Knowledge Center:

  • IBM Cloud Pak System 2.3.3.0 Intel based IBM Cloud Pak System models W2500, W3500 and W3550 are supported. There is currently no support for the Power based IBM Cloud Pak System model W3700.
  • IBM OS image for Red Hat Linux Systems (RHEL 7.7 X64) Version 7.7 Scenarios using a custom OS image are also supported, as long as it is Red Hat Enterprise Linux (RHEL) 7.7 or higher.
  • IBM OS image for Red Hat Enterprise Linux CoreOS Version 4.4.0 This is included with accelerator bundle for OpenShift Container Platform 4.4.0.0.
  • Docker Accelerator 1.0.13.0 Docker registry is required for accessing OpenShift Container Platform images. Obtain Docker registry accelerator 1.0.13.0 from IBM FixCentral.
  • OpenShift Container Platform Accelerator 4.4.0.0 OpenShift Container Platform accelerator is used for deploying OpenShift Container Platform cluster in IBM Cloud Pak System. Get OpenShift Container Platform accelerator 4.4.0.0 from IBM FixCentral.
  • Red Hat Satellite Server 6 shared service deployed The shared service should be connected to an existing Red Hat Satellite Server (RHSS), or to RHSS deployed on IBM Cloud Pak System with access to repositories rhel-7-server-rpms & rhel-7-server-extras-rpms.

Note that IBM Cloud Pak System comes with Red Hat subscriptions for RHEL and RHSS.

  • Active subscription with Red Hat for the OpenShift Container Platform Unlike the Red Hat subscription for RHEL and RHSS, the OpenShift Container Platform (OCP) subscription is not included with IBM Cloud Pak System. Please refer to OpenShift Container Platform accelerator V4.3.1.0 recipe for detailed steps.
  • Sufficient compute, memory and storage resources on IBM Cloud Pak System A single OpenShift Container Platform cluster requires at least 28 virtual CPUs, 112 GB of RAM, and 1202 GB of storage.

For detailed prerequisite steps to configure Red Hat Satellite Server and Private Docker Registry on IBM Cloud Pak System, refer to OpenShift Container Platform accelerator V4.3.1.0 recipe.

Step-by-step

  1. Load and verify required artefacts on IBM Cloud Pak System V2.3.3.0

    By default, IBM Cloud Pak System 2.3.3.0 includes most of the required content pre-loaded for you.

    For the sake of completeness, see the following complete list of everything that you need:

    Content artefacts  Type  IBM Fix Central link
     IBM OS Image Red Hat Linux Enterprise Server V3.1.0.0 VM  Virtual Image IBM_OS_Image_RedHat_LS_V3.1.0.0_VM-cps
     Foundation Pattern Type V2.1.17.0  Pattern Type foundation-2.1.17.0-cps
     Red Hat OS Update Service V1.0.15.0  Pattern Type rhus-1.0.15.0-cps
     Docker accelerator V1.0.13.0  Pattern Type docker-1.0.13.0-cps
     IBM Cloud Pak System accelerator bundle for Red Hat OpenShift V4.4.0  BYOL Binaries cps-openshift-4-4-0-intel-content.tar.gz
  2. Verifying and importing BYOL binaries

    The “IBM Cloud Pak System accelerator bundle for Red Hat Openshift V4.4.0” are not installed by default. To confirm, log in to IBM Cloud Pak System and go to System > Storehouse Browser. If you do not see an entry for /admin/files/RedHatOpenShift as shown in Figure 1, then it means that the binaries are not installed yet.

     Figure 1

    Figure 1

    Process for loading the binaries has been documented here in the IBM Cloud Pak System Knowledge Center.

    /****************************************************
    Cloud Pak Accelerator Bundle Import utility
    ***************************************************
    Artifacts from accelerator Bundle will be imported to Cloud Pak System
    Cloud Pak System hostname or IP address: 9.XX.XX.XX
    Cloud Pak System username: admin
    Password for admin:
    - Testing connectivity to Cloud Pak System
    Upload Cloud Pak binaries to Storehouse
    -----------------------------------------------
    - Uploading binaries for: Redhat Openshift 4.4.6 on IBM Cloud Pak System
    - Pre-upload Verification
    1) openshift-install-linux-4.3.1.tar.gz: verified successfully
    2) ocp4.4.6-x86_64.tgz: verified successfully
    3) openshift-install-linux-4.4.6.tar.gz: verified successfully
    4) openshift-client-linux-4.4.6.tar.gz: verified successfully
    5) ocp4.3.1-x86_64.tgz: verified successfully
    6) ocp4.3.1-x86_64-extra.tar: verified successfully
    7) jq-linux64: verified successfully
    8) openshift-client-linux-4.3.1.tar.gz: verified successfully
    - All binaries verified successfully.
    - openshift-install-linux-4.3.1.tar.gz is already on the server
    - ocp4.4.6-x86_64.tgz is already on the server
    - openshift-install-linux-4.4.6.tar.gz is already on the server
    - openshift-client-linux-4.4.6.tar.gz is already on the server
    - ocp4.3.1-x86_64.tgz is already on the server
    - ocp4.3.1-x86_64-extra.tar is already on the server
    - jq-linux64 is already on the server
    - openshift-client-linux-4.3.1.tar.gz is already on the server
    - Verifying file
    Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
    - Verifying file
    Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
    - Downloading CLI from Cloud Pak System
    - Starting Cloud Pak System CLI
    OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/jna8015534376793836385.tmp which might have disabled stack guard. The VM will try to fix the stack guard now.
    It’s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack’.
    Import and Clone Virtual Images
    ----------------------------------------
    - Checking for Virtual Image
    Name: IBM OS Image for Red Hat Linux Systems
    Version: 3.1.0.0
    - Virtual Image is available
    - Working with image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
    Version: 4.3.0
    - Image exists. Skipping importing the image
    - Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
    - Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 250G
    - Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
    - Working with image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 16G
    Version: 4.4.3
    - Image exists. Skipping importing the image
    - Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 120G
    - Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS - 250G
    - Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
    Import Ptypes
    ------------------------
    - Import ptype for: Redhat Openshift 4.4.6 on IBM Cloud Pak System
    - Working with ptype
    Name: openshift
    Version: 4.4.0.0
    - Ptype exists. Skipping the Ptype import*/

    Confirm that the following files are now visible from the IBM Cloud Pak System. Go to System > Storehouse browser and confirm that you see what is shown in Figure 2.

    CPS-OCP-BYOL-Imported

    Figure 2

    New and improved Cloud Pak accelerator bundle import utility is available for OpenShift Container Platform V4.4.0.0 accelerator on IBM Cloud Pak System. It loads images to storehouse, clones the Red Hat Enterprise Linux CoreOS OVA to three variants with different disk size of 16 GB, 120 GB, and 250 GB, and makes them available in IBM Cloud Pak System Virtual Images catalog as shown in Figure 3. In addition, it also imports the accelerator into IBM Cloud Pak System.

    CPS-VirtualImages

    Figure 3

  3. Verifying and importing Pattern Types

    The new and improved Cloud Pak accelerator bundle import utility (described in previous step) for IBM Cloud Pak System loads pattern type to IBM Cloud Pak System too. After it is loaded, you should see the OpenShift 4.4.0.0 accelerator in the catalog with status available as shown in Figure 4.

    CPS-AcceleratorAvailable

    Figure 4

  4. Exploring Red Hat OpenShift 4 cluster accelerator

    Navigate to Provision environments from IBM Cloud Pak System user interface as shown in Figure 5.

    IBM Cloud Pak System home page

    Figure 5

    Search for OpenShift Container Platform and click Ready to deploy as shown in Figure 6.

    OpenShift Container Platform.

    Figure 6

    This action opens a page to select Default or Customized cluster.

    Select Customized as shown in Figure 7 and click Continue. The Configure deployment page opens.

    OpenShift Container Platform - customized deployment 

    Figure 7

    The first section in this page includes IBM Cloud Pak System deployment options as shown in Figure 8. For example, environment profile, cloud group, and so on. Select the appropriate values such that all prerequisites are available in the chosen cloud group.

    OpenShift Container Platform - environment details

    Figure 8

    The second section in this page contains OS users root and virtuser credentials for helper virtual machines as shown in Figure 9.

    OpenShift Container Platform - credentials.

    Figure 9

    The last section in this page includes an optional parameter to generate SSH key, which is used in deployment as shown in Figure 10. If you generate this key, you can use it to connect to helper and OpenShift Container Platform virtual machines via SSH.

    OpenShift Container Platform - optional parameters

    Figure 10

    Click Configure nodes to open the Configure nodes tab page.

    This page shows type of VMs, such as helpers, control planes and worker nodes, which are included in the accelerator and hardware resource configuration for each virtual machine as shown in Figure 11 and Figure 12. You can change the resource configuration of all nodes in a cluster to be deployed. Use slider, as shown in Figure 11, to set the desired value for each resource type of virtual machines.

    OpenShift Container Platform - hardware allocation

    Figure 11

    Note: You cannot change number of masters to be deployed as Red Hat recommends deploying three masters for OpenShift Container Platform 4 clusters.

    OpenShift Container Platform - nodes configuration

    Figure 12

    Click OpenShift options to open the OpenShift options tab page.

    This page shows deployment parameters for OpenShift Container Platform accelerator as shown in Figures 13 and 14. Detailed description of parameters can be found here.

    OpenShift version

    It is the Red Hat OpenShift Container Platform version to deploy.

    OpenShift cluster domain name

    It is the domain name for Red Hat OpenShift Container Platform cluster. If not specified, then a default cluster domain name is generated.

    OpenShift cluster name

    It is the Red Hat OpenShift Container Platform cluster name. If not specified, then a default cluster name is generated.

    OpenShift image registry name

    It is the fully qualified name for the Red Hat OpenShift Container Platform image registry. In case there is no Cloud Pak System Registry available in cloud group or you wish to use a different docker registry for accessing OpenShift Container Platform images, specify registry hostname:port here.

    OpenShift image registry username

    It is the username needed to access the Red Hat OpenShift Container Platform secure image registry.

    OpenShift image registry password

    It is the user password that is needed to access the Red Hat OpenShift Container Platform secure image registry.

    OpenShift pull-secret

    If access to external OpenShift registry is available and you wish to use it for OpenShift Container Platform images, paste contents of the downloaded OpenShift install pull-secret.json file here.

    OpenShift Container Platform - parameters

    Figure 13

    Alternate NFS server name or IP address (for the OpenShift image registry)

    It is the alternate NFS server name or IP address of the OpenShift image registry. By default, Primary helper virtual machine is used to configure storage for OpenShift image registry. If this parameter value is specified, then this external NFS server is used for OpenShift image registry instead. Please see this page about configuring NFS server and path.

    Alternate NFS server path (for the OpenShift image registry)

    It is the alternate NFS server path of the OpenShift image registry. It is used along with the Alternate NFS server name or IP address parameter to configure storage for OpenShift image registry.

    Alternate NFS server name or IP address (for the application persistent storage)

    It is the alternate NFS server name or IP address of the Application Persistent Storage. By default Primary helper is used to configure persistent storage for workloads to be deployed on OpenShift Container Platform cluster. When value is specified here, it is used for persistent storage for workloads instead.

    Alternate NFS server path (for the application persistent storage)

    It is the alternate NFS server path of the Application Persistent Storage. It is used along with Alternate NFS server name or IP address parameter to configure persistent storage for workloads.

    OpenShift Container Platform - alternate NFS parameters

    Figure 14

    Click Review and deploy to open the Ready to deploy tab page. This page shows a summary of topology and resource configuration as shown in Figure 15.

    OpenShift Container Platform - topology summary

    Figure 15

    Following is a brief description of various nodes included in cluster.

    PrimaryHelper and SecondaryHelper

    There are two Helper nodes that run on RHEL 7.7. It supports the deployment of OpenShift cluster on virtual machines that run Red Hat Core OS. The Helper node provide services for the OpenShift Cluster as documented in the OpenShift 4 documentation: Helper Git Repository and Helper Blog. IBM implements two Helper virtual machines and uses a floating IP address to provide high availability for these services.

    Bootstrap

    There is one Bootstrap node that is used to install the OpenShift Container Platform control plane on the Master nodes. It is only used during the bootstrapping of OpenShift Container Platform. Eventually, this virtual machine is destroyed and resources are released.

    Master

    There are three Master nodes deployed on virtual machines that run on Red Hat Core OS. OpenShift 4 requires three Master nodes, ensuring high availability and quorum of essential Kubernetes services like etcd.

    Worker

    By default, there are two Worker nodes deployed on virtual machines that run on Red Hat Core OS. This ensures high availability of containers that run on these Worker nodes. Depending on the needs for your OpenShift cluster, you could opt for a higher number of Worker nodes or Worker nodes with more CPU and memory. Starting from IBM Cloud Pak System V2.3.3.0, it is possible to add additional Worker nodes to your OpenShift cluster after deployment (horizontal scaling).

    As shown in the following table, by default, single OCP cluster requires 28 virtual CPUs, 112 GB of RAM and 1202 GB of storage. Depending on the number and sizing of the worker nodes, the amount of resources required could be higher.

     

     VM  Number  OS  virtual CPUs  RAM (GB)  storage (GB)
     Primary Helper  1  RHEL 7.7  4  16  470
     Secondary Helper  1  RHEL 7.7  4  16  12
     Bootstrap  1  RH Core OS  4  16  120
     Master  3  RH Core OS  4  16  120
     Worker  2  RH Core OS  2  8  120
     Total  8 28 112 1202
  5. Deployment of Red Hat OpenShift 4 cluster

    With all the previous steps completed, you are now ready to deploy your first Red Hat OpenShift 4.4 cluster!

    Go to Cloud Pak System user interface (https://<ICPS_system_IP>/cps/) and log in. The Getting started page opens as shown in Figure 16.
    IBM Cloud Pak System - home

    Figure 16

    Click Provision environment as shown in Figure 17.
    IBM Cloud Pak System - Navigate to provision environment page

    Figure 17

    The Provision environment page opens as shown in Figure 18.

    IBM Cloud Pak System - provision environment page

    Figure 18

    Search for OpenShift Container Platform and click Ready to deploy as shown in Figure 19.
    OpenShift Container Platform - ready to deploy

    Figure 19

    User can deploy OpenShift Container Platform with default or customized configuration

    You should see a page as shown in Figure 20. By default, OpenShift Container Platform version is set to 4.4.

    OpenShift Container Platform - select default or customized deployment

    Figure 20

    Select Default or Customized option based on your requirement.

    Default deployment

     

    Default deployment option deploys an OpenShift Container Platform cluster with already configured values with minimum hardware OpenShift Container Platform cluster. To deploy default cluster, select Default as shown in Figure 21.

    OpenShift Container Platform - select default

    Figure 21

    Customized deployment

    Customized deployment enables you to configure the cluster as per business use case. It provides a way to change hardware specification for the OpenShift Container Platform cluster to be deployed. To deploy Cutomized cluster select Customized as shown in Figure 22 and continue.

    OpenShift Container Platform - select customized

    Figure 22

    If you want the customize deployment steps, refer to section Exploring Red Hat OpenShift 4 cluster accelerator section of this article. If you want to do a default deployment, continue with this procedure.

    Click Continue to open the Configure deployment page as shown in Figure 23.

    First section on this page shows environment profile, cloud group, and IP group for deployment. Select appropriate values based on where the RHUS and Cloud Pak System registry shared services are running.
    OpenShift Container Platform - default environment details.

    Figure 23

    In Helper node credentials section, specify password for root and virtuser as shown in Figure 24.

    OpenShift Container Platform - default credentials

    Figure 24

    Optionally, in the Optional section as shown in Figure 25, specify a SSH key to connect to helper virtual machines via SSH post deployment. If not specified, then a default key is generated and used.
    OpenShift Container Platform - default - optional configuration

    Figure 25

    Click Review and deploy as shown in Figure 26.
    OpenShift Container Platform - default - click review and deploy

    Figure 26

    Review cluster topology and other values.

    Click deploy as shown in Figure 27 to deploy cluster.
    OpenShift Container Platform - default - click deploy

    Figure 27

    Within seconds you should see a message indicating that the deployment has started as shown in Figure 28. 10. In the message box, click Manage environments. You will be redirected to Manage environments page.

     OpenShift Container Platform - default - go to manage environments

    Figure 28

    As shown in figure 29, deployment starts with the cluster status in launching state.

    OpenShift Container Platform - manage environments

    Figure 29

    It takes approximately 50 minutes to deploy OpenShift Container Platform cluster. After it is deployed, you should see an instance as shown in Figure 30.

     OpenShift Container Platform - deployment completed

    Figure 30

    Review history section of the instance for post deployment actions as highlighted in Figure 31.
     OpenShift Container Platform - review history

    Figure 31

  6. Post deployment actions

    Before you can use the OpenShift 4 cluster, a few more steps are required as documented in step 6 of Getting started with OpenShift Container Platform 4.x pattern.

    Review history section for next steps as indicated in Figure 30.

    A. Retrieve the password for kubeadmin

    The kubeadmin password gets generated during the installation of OpenShift 4. Retrieve kubeadmin password by providing root password for Helper VM as shown in Figure 32.

    OpenShift Container Platform - Retrieve kubeadmin password

    Figure 32

    B. Configure your DNS server

    Set up the following two DNS wildcard entries for the floating IP address and fully-qualified domain name of your OpenShift 4 Virtual System Instance. This is required to access the OpenShift web-console, applications, and APIs.

    *.<fqdn> IN A <ip>
    *.apps.<fqdn> IN A <ip>

    In the case of our OpenShift 4 cluster here, the floating IP address is ocp_cluster_ip with corresponding fully-qualified domain name cps-rack-79-vm-12.rtp.raleigh.ibm.com. So you need to configure the following DNS wildcard entries:

    *.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
    *.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15

    If you are unable to easily make changes to your DNS server, you can add the following entries to your local /etc/hosts file (or equivalent on Windows) for testing purposes. This will allow you to log in to the OpenShift console, but note that you would need additional entries for any applications you would deploy later.

    ocp_cluster_ip console-openshift-console.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx oauth-openshift.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx

    You can find more information about OpenShift external DNS requirements here. The DNS records listed as “This record must be resolvable by both clients external to the cluster …” are required. DNS is also provided on the Helper Nodes to cover the resolution inside the cluster.

    If you are able to configure DNS records up front, then the cluster console link will be accessible immediately. Configuring DNS ahead of time is the recommended approach for deploying OpenShift Container Platform clusters on Cloud Pak System. You would need to create the following records in your DNS server for each IP in the IP group you are using to deploy (so that any IP that is selected from the IP group to be the floating IP for the cluster will already have wildcard entries associated with it in DNS):

    *.sub.domain IN A <ip>
    *.mycluster.sub.domain IN A <ip>
  7. Access your OpenShift cluster

    You can now access your OpenShift 4 cluster using the OpenShift console link as shown in Figure 33.

    Access OCP console

    Figure 33

    Log in with the username kubeadmin and the password that you retrieved earlier as shown in Figure 34.

    Login to OCP console

    Figure 34

    After you log in, you will see the console as shown in Figure 35.

    Verify OCP console

    Figure 35

    Navigate to Compute > Nodes. If you see three Master nodes and two Worker nodes, then it confirms that the OpenShift 4 cluster topology was deployed as expected.

  8. Verify your OpenShift cluster

    Inspect your cluster from IBM Cloud Pak Console -> Manage environments

    A. Verify topology

    In the IBM Cloud Pak Console -> Manage environments page, click Nodes for the deployed instance as shown in Figure 36, you can verify cluster topology and VMs deployed as part of instance.

    Cluster topology

    Figure 36

    B. Verify middleware roles

    In the IBM Cloud Pak Console -> Manage environments, click Middleware for the deployed instance as shown in Figure 37. You can verify the cluster topology and deployed VMs as a part of the instance.

    Middleware view

    Figure 37

    NOTE: Do not stop or restart OpenShift Container Platform cluster or any nodes until after 24 hours of deployment. Failing to do so may render your cluster to a broken state that cannot be recovered. Refer this page for more details.

  9. Register your OpenShift cluster with Red Hat

    Finally, do not forget to register your OpenShift cluster with Red Hat. This manual step is required if your OpenShift cluster does not have internet access to reach Red Hat. You can follow step 4 here to register your cluster on the “Cluster registration” page.

  10. Next steps

    Now you are ready for day 2 operations and workload deployment on your OpenShift Container Platform cluster. See this article for day 2 OpenShift Container Platform cluster operations in IBM Cloud Pak System.

  11. Conclusion

    IBM Cloud Pak System 2.3.3.0 enables clients to quickly roll out one or more Red Hat OpenShift 4.4 clusters, which greatly simplifies the process, ensures consistency, and avoids human error. This is also used as the foundation for the deployment of IBM Cloud Paks on the IBM Cloud Pak System platform.

    I would like to thank fellow IBMers Hina Sharma, Shreya Kunar and Shyamala Rajagopalan for their help in creating this tutorial.

Join The Discussion