Overview

Skill Level: Intermediate

This blog describes OpenShift Container Storage 4.5 deployment on IBM Cloud Pak System using Red Hat OpenShift Container Platform 4.6 and local storage.

Ingredients

Before you deploy your first OpenShift Container Platform 4.6 cluster with OpenShift Container Storage on IBM Cloud Pak System, few prerequisites need to be in place. For more information about the prerequisites, see IBM Knowledge Center.

IBM Cloud Pak System 2.3.3.3

Intel based IBM Cloud Pak System models W2500, W3500, W3550 are supported. There is currently no support for the Power based IBM Cloud Pak System model W3700.

IBM OS image for Red Hat Linux Systems (RHEL 7.8 X64) Version 7.8

Scenarios using a custom OS image are also supported as long as it is Red Hat Enterprise Linux (RHEL) 7.8 or higher.

IBM OS image for Red Hat Enterprise Linux CoreOS Version 4.6.1 

This is included with accelerator bundle for OpenShift Container Platform 4.6.0.0.

OpenShift Container Platform Accelerator 4.6.0.0 

OpenShift Container Platform accelerator is used for deploying OpenShift Container Platform cluster in IBM Cloud Pak System. Get OpenShift Container Platform accelerator 4.6.0.0 from IBM Fix Central.

OpenShift Container Storage Accelerator 4.5

OpenShift Container Storage accelerator is used for deploying OpenShift Container Storage cluster in IBM Cloud Pak System. Get OpenShift Container Storage accelerator 4.5 from IBM Fix Central.

Red Hat Satellite Server 6 shared service deployed

The shared service must be connected to an existing Red Hat Satellite Server (RHSS) or to RHSS deployed on IBM Cloud Pak System with access to repositories rhel-7-server-rpms & rhel-7-server-extras-rpms. Note that IBM Cloud Pak System comes with Red Hat subscriptions for RHEL and RHSS.

Active subscription with Red Hat for the OpenShift Container Platform

Unlike the Red Hat subscription for RHEL and RHSS, the OpenShift Container Platform (OCP) subscription is not included with IBM Cloud Pak System.*

Sufficient compute, memory and storage resources on IBM Cloud Pak System

OpenShift Container Storage 4.5 is deployed as a minimal cluster of three worker nodes. It may be expanded in sets of three nodes to a maximum of 50 worker nodes. Each node supports one to three 2 TiB disks for storage, with the storage from each node replicated to two others within the system. Storage used by OpenShift Container Storage is obtained using a Persistent Volume Claim (PVC) and Persistent Volume (PV) from local block storage or from default thin storage class in case of VMware system.

Each node on which OpenShift Container Storage runs require 16 vCPUs and 64 GB of memory. These nodes are in addition to the worker nodes that are used to run application pods.

A single OpenShift Container Platform with OpenShift Container Storage cluster and default configuration in IBM Cloud Pak System requires atleast 76 virtual CPUs, 304 GB of RAM and 1820 GB of storage.

Step-by-step

  1. Loading and validating content loaded on IBM Cloud Pak System 2.3.3.3

    By default, most of the content is preloaded for you in IBM Cloud Pak System 2.3.3.3.

    For the sake of completeness, see the following list of everything that you need:

    Content artifact Type IBM Fix Central link
    IBM OS Image Red Hat Linux Enterprise Server V3.1.2.0 VM Virtual Image IBM_OS_Image_RedHat_LS_V3.1.2.0_VM-cps
    Foundation Pattern Type V2.1.18.0 Pattern Type foundation-2.1.18.0-cps
    Red Hat OS Update Service V1.0.16.0 Pattern Type rhus-1.0.16.0-cps
    Docker accelerator V1.0.14.0 Pattern Type docker-1.0.14.0-cps

    IBM Cloud Pak System accelerator bundle for Red Hat OpenShift V4.6.0

    BYOL Binaries cps-openshift-4-6-0-intel-content.tar.gz

     

     A. Verifying and importing BYOL binaries

    The “IBM Cloud Pak System accelerator bundle for Red Hat Openshift V4.6.0” is not installed by default. To confirm, do the following steps:

    • Log in to IBM Cloud Pak System.
    • Go to System > Storehouse Browser.
    • Check whether you can see an entry for /admin/files/RedHatOpenShift as shown in Figure 1. If you do not see the entry, then it means that the binaries are not installed yet. 

    IBM Cloud Pak System accelerator bundle for Red Hat OpenShift V4.6.0 have not been loaded

     Figure 1: IBM Cloud Pak System accelerator bundle for Red Hat OpenShift V4.6.0 have not been loaded

    Process for loading the binaries has been documented here in the IBM Cloud Pak System Knowledge Center.

     ***************************************************

    Cloud Pak Accelerator Bundle Import utility
    ***************************************************
    Artifacts from accelerator Bundle will be imported to Cloud Pak System
    Cloud Pak System hostname or IP address: 9.XX.XX.XX
    Cloud Pak System username: admin
    Password for admin:
    – Testing connectivity to Cloud Pak System
    Upload Cloud Pak binaries to Storehouse
    ———————————————–
    – Uploading binaries for: Redhat Openshift 4.4.6 on IBM Cloud Pak System
    – Pre-upload Verification
    1) openshift-install-linux-4.3.1.tar.gz: verified successfully
    2) ocp4.4.6-x86_64.tgz: verified successfully
    3) openshift-install-linux-4.4.6.tar.gz: verified successfully
    4) openshift-client-linux-4.4.6.tar.gz: verified successfully
    5) ocp4.3.1-x86_64.tgz: verified successfully
    6) ocp4.3.1-x86_64-extra.tar: verified successfully
    7) jq-linux64: verified successfully
    8) openshift-client-linux-4.3.1.tar.gz: verified successfully
    – All binaries verified successfully.
    – openshift-install-linux-4.3.1.tar.gz is already on the server
    – ocp4.4.6-x86_64.tgz is already on the server
    – openshift-install-linux-4.4.6.tar.gz is already on the server
    – openshift-client-linux-4.4.6.tar.gz is already on the server
    – ocp4.3.1-x86_64.tgz is already on the server
    – ocp4.3.1-x86_64-extra.tar is already on the server
    – jq-linux64 is already on the server
    – openshift-client-linux-4.3.1.tar.gz is already on the server
    – Verifying file
    Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
    – Verifying file
    Name: IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
    – Downloading CLI from Cloud Pak System
    – Starting Cloud Pak System CLI
    OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/jna8015534376793836385.tmp which might have disabled stack guard. The VM will try to fix the stack guard now.
    It’s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack’.
    Import and Clone Virtual Images
    —————————————-
    – Checking for Virtual Image
    Name: IBM OS Image for Red Hat Linux Systems
    Version: 3.1.0.0
    – Virtual Image is available
    – Working with image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS – 16G
    Version: 4.3.0
    – Image exists. Skipping importing the image
    – Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS – 120G
    – Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS – 250G
    – Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_16G_4_3_0.ova
    – Working with image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS – 16G
    Version: 4.4.3
    – Image exists. Skipping importing the image
    – Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS – 120G
    – Cloned Image exists. Skipping cloning of image
    Name: IBM OS Image for Red Hat Enterprise Linux CoreOS – 250G
    – Image import completed for IBM_OS_Image_for_Red_Hat_Enterprise_Linux_CoreOS_-_16G.ova
    Import Ptypes
    ————————
    – Import ptype for: Redhat Openshift 4.4.6 on IBM Cloud Pak System
    – Working with ptype
    Name: openshift
    Version: 4.6.0.0
    – Ptype exists. Skipping the Ptype import 

    IBM Cloud Pak System accelerator bundle for Red Hat OpenShift V4.6.0 have been loaded

     Figure 2: IBM Cloud Pak System accelerator bundle for Red Hat OpenShift V4.4.0 have been loaded

    New and improved Cloud Pak accelerator bundle import utility is available for OpenShift Container Platform V4.6.0.0 accelerator on IBM Cloud Pak System. It loads images to storehouse, clones the Red Hat Enterprise Linux CoreOS OVA to three variants with different disk size of 16 GB, 120 GB, and 250 GB, and makes them available in IBM Cloud Pak System Virtual Images catalog as shown in Figure 3. In addition, it also imports the accelerator into IBM Cloud Pak System. 

    Imported and cloned Core OS Virtual Image is now available in catalog

    Figure 3: Imported and cloned Core OS Virtual Image is now available in catalog

    B. Verifying and importing Pattern Types

    The new and improved Cloud Pak accelerator bundle import utility (described in previous step) for IBM Cloud Pak System loads pattern type to IBM Cloud Pak System too. After it is loaded, you should see the OpenShift Container Platform accelerator 4.6.0.0 in the catalog with status available as shown in Figure 4. 

    OpenShift 4.4.0.0 accelerator loaded in the catalog with status available

    4: OpenShift Container Platform accelerator 4.6.0.0 loaded in the catalog with status available

  2. Exploring Red Hat OpenShift Container Platform with OCS accelerator

    1. Navigate to Provision accelerators from IBM Cloud Pak System user interface as shown in Figure 5.

     Figure_5-1

    Figure 5: IBM Cloud Pak System home page

    2. Search for OpenShift Container Platform with OCS.

    3. Check if it is in Ready to deploy state and click the OpenShift Container Platform with OCS tile as shown in Figure 6.

    OpenShift Container Platform with OCS

     Figure 6: OpenShift Container Platform with OCS.

    This action opens a page to select Default or Customized cluster.

    4. Select Customized as shown in Figure 7 and click Continue. The Configure deployment page opens. 

    OpenShift Container Platform with OCS – customized deployment

    Figure 7: OpenShift Container Platform with OCS – customized deployment

     The first section in this page includes IBM Cloud Pak System deployment options as shown in Figure 8. For example, environment profile, cloud group, and so on. Select the appropriate values such that all prerequisites are available in the chosen cloud group. 

    OpenShift Container Platform with OCS – environment details

     Figure 8: OpenShift Container Platform with OCS – environment details

     The second section in this page contains OS users, such as root and virtuser credentials for helper virtual machines as shown in Figure 9. Also, provide value for OpenShift pull-secret, which is the content of OpenShift install pull-secret.json file. You would have downloaded the json file from Red Hat site. It pulls the required images for installing OpenShift Container Storage and OpenShift Container Platform. 

     OpenShift Container Platform with OCS – credentials

     Figure 9: OpenShift Container Platform with OCS – credentials

     The last section in this page includes an optional parameter to generate SSH key, which is used in deployment as shown in Figure 10. If you generate this key, you can use it to connect to helper and OpenShift Container Platform virtual machines via SSH. 

    OpenShift Container Platform with OCS – optional parameters

     Figure 10: OpenShift Container Platform with OCS – optional parameters

     5. Click Configure nodes to open the Configure nodes tab page.

    This page shows type of VMs, such as helpers, control planes, worker nodes, which are included in the accelerator and hardware resource configuration for each virtual machine as shown in Figure 11 and Figure 12. You can change the resource configuration of all nodes in a cluster to be deployed. Use slider, as shown in Figure 11, to set the desired value for each resource type of virtual machines. 

    OpenShift Container Platform with OCS – hardware allocation

     Figure 11: OpenShift Container Platform with OCS – hardware allocation

     Note: You cannot change the number of masters to be deployed as Red Hat recommends deploying three masters for OpenShift Container Platform 4 clusters. 

    OpenShift Container Platform with OCS – nodes configuration

    Figure 12: OpenShift Container Platform with OCS – nodes configuration

     6. Click OpenShift options to open the OpenShift options tab page.

    This page shows deployment parameters for OpenShift Container Platform as shown in Figures 13 and 14. Detailed description of parameters can be found here.

    OpenShift version: It is the Red Hat OpenShift Container Platform version to deploy.

    OpenShift cluster domain name: It is the domain name for Red Hat OpenShift Container Platform cluster. If not specified, then a default cluster domain name is generated.

    OpenShift cluster name: It is the Red Hat OpenShift Container Platform cluster name. If not specified, then a default cluster name is generated.

    OpenShift Container Platform with OCS – parameters

     Figure 13: OpenShift Container Platform with OCS – parameters

    7. Click Review and deploy to open the Ready to deploy tab page. This page shows a summary of topology and resource configuration as shown in Figure 14.

     OpenShift Container Platform with OCS – topology summary

    Figure 14: OpenShift Container Platform with OCS – topology summary

    PrimaryHelper and SecondaryHelper

    There are two Helper nodes that run on RHEL 7.8. It supports the deployment of OpenShift cluster on virtual machines that run Red Hat Core OS. The Helper node provide services for the OpenShift Cluster as documented in the OpenShift 4 documentation: Helper Git Repository and Helper Blog . IBM implements two Helper virtual machines and uses a floating IP address to provide high availability for these services.

    Bootstrap
    There is one Bootstrap node that is used to install the OpenShift Container Platform control plane on the Master nodes. It is only used during the bootstrapping of OpenShift Container Platform. Eventually, this virtual machine is destroyed and resources are released.

    Master
    There are three Master nodes deployed on virtual machines that run on Red Hat Core OS. OpenShift 4 requires three  Master nodes, ensuring high availability and quorum of essential Kubernetes services like etcd.

    Worker
    By default, there are five worker nodes deployed on virtual machines that run on Red Hat Core OS for OpenShift Container Platform with OCS. This ensures high availability of containers that run on these worker nodes.

    As shown in the following table, by default, single OCP cluster with OCS requires 76 virtual CPUs, 304 GB of RAM and 1820 GB of storage. Depending on the number and sizing of the worker nodes, the amount of resources required could be higher. 

    VM Number OS virtual CPUs RAM (GB) storage (GB)
    Primary Helper 1 RHEL 7.8 4 16 170
    Secondary Helper 1 RHEL 7.8 4 16 60
    Bootstrap 1 RH Core OS 4 16 60
    Master 3 RH Core OS 4 16 0
    Worker 2 RH Core OS 2 8 0
    Storage worker 3 RH Core OS 16 64 510
    Total 11 76 304 1820
  3. Deployment of Red Hat OpenShift Container Platform with OCS cluster

    With all the previous steps completed, you are now ready to deploy your first Red Hat OpenShift Container Platform with OCS cluster!

    1. Go to Cloud Pak System user interface (https://<ICPS_system_IP>/cps/) and log in. The Getting started page opens as shown in Figure 15.

     

    IBM Cloud Pak System – home

     

    Figure 15: IBM Cloud Pak System – home

    2. Click Provision accelerators as shown in Figure 16.

    IBM Cloud Pak System – Navigate to provision environment page

     

    Figure 16: IBM Cloud Pak System – Navigate to provision accelerators page

    The Provision accelerators page opens as shown in Figure 17. 

    IBM Cloud Pak System – provision environment page

     Figure 17: IBM Cloud Pak System – provision accelerators page

    3. Search for OpenShift Container Platform with OCS.

    4. Check whether the accelerator is in Ready to deploy state and click the OpenShift Container Platform with OCS tile as shown in Figure 18.

     

    OpenShift Container Platform with OCS – ready to deploy

     Figure 18: OpenShift Container Platform with OCS – ready to deploy

    You can deploy OpenShift Container Platform with OCS with default or customized configuration.

    You should see a page as shown in Figure 19. By default, OpenShift Container Platform version is set to 4.6.

    Figure_7

     Figure 19: OpenShift Container Platform with OCS – select default or customized deployment

    5. Select Default or Customized option based on your requirement.

    • Default deployment

          Default deployment option deploys an OpenShift Container Platform with OCS cluster. It comes with configured values and minimum hardware requirements for OpenShift Container Platform with OCS cluster. To deploy default cluster, select Default in Figure 20.

     OpenShift Container Platform with OCS – select default

     

    • Customized deployment

    Customized deployment enables you to configure the cluster as per business use case. It provides a way to change hardware specification for the OpenShift Container Platform with OCS cluster to be deployed. To deploy Cutomized cluster select Customized as shown in Figure 21 and continue. 

    OpenShift Container Platform with OCS – select customized

    Figure 21: OpenShift Container Platform with OCS – select customized

    If you want the customize deployment steps, refer to section Exploring Red Hat OpenShift Container Platform with OCS accelerator section of this article. If you want to do a default deployment, continue with this procedure.

    6. Click Continue to open the Configure deployment page as shown in Figure 22.

        First section on this page shows environment profile, cloud group, and IP group for deployment. Select appropriate values based on where the RHUS shared service is running. 

    OpenShift Container Platform with OCS – default environment details.

     Figure 22: OpenShift Container Platform with OCS – default environment details.

     7. In Helper node credentials section, specify password for root, virtuser and specify pull-secret as shown in Figure 23. 

    OpenShift Container Platform with OCS – default credentials

     Figure 23: OpenShift Container Platform with OCS – default credentials

    8. Optionally, in the Optional section, specify a SSH key to connect to helper virtual machines via SSH post deployment. If not specified, then a default key is generated and used.

     

    OpenShift Container Platform with OCS – default – optional configuration

     

    Figure 24: OpenShift Container Platform with OCS – default – optional configuration

    9. Click Review and deploy as shown in Figure 25.

     

    OpenShift Container Platform with OCS – default – click review and deploy

     

    Figure 25: OpenShift Container Platform with OCS – default – click review and deploy

    10. Review cluster topology and other values.

    11. Click Deploy as shown in Figure 26 to deploy the cluster.

     

    OpenShift Container Platform with OCS – default – click deploy

     

    Figure 26: OpenShift Container Platform with OCS – default – click deploy

    Within seconds, you should see a message indicating that the deployment has started as shown in Figure 27.

    12. In the message box, click Manage accelerator instances. You will be redirected to Manage accelerator instances page.

     

    Figure_27-1

     Figure 27: OpenShift Container Platform with OCS – default – go to Manage accelerator instances

    It takes approximately 50 minutes to deploy OpenShift Container Platform with OCS cluster. After it is deployed, you should see an instance as shown in Figure 28.

    13. Review history section of the instance for post deployment actions.

     

    OpenShift Container Platform with OCS – deployment completed

     

     Figure 28: OpenShift Container Platform with OCS – deployment completed

  4. Post deployment actions

    Before you can use the OpenShift Container Platform with OCS cluster, a few more steps are required as documented in step 6 of Getting started with OpenShift Container Platform 4.x pattern.

     Review history section for next steps as indicated in Figure 28.

    A. Retrieve the password for kubeadmin

    The kubeadmin password gets generated during the installation of OpenShift 4. Retrieve kubeadmin password by providing root password for *Helper VM_ as shown in Figure 29. 

    OpenShift Container Platform – Retrieve kubeadmin password

     Figure 29: OpenShift Container Platform – Retrieve kubeadmin password

     B. Configure your DNS server

    Set up the following two DNS wildcard entries for the floating IP address and fully-qualified domain name of your OpenShift 4 Virtual System Instance. This is required to access the OpenShift web-console, applications, and APIs.

    *.<fqdn> IN A <ip>
    *.apps.<fqdn> IN A <ip>
    In the case of our OpenShift 4 cluster here, the floating IP address is ocp_cluster_ip with corresponding fully-qualified domain name cps-rack-79-vm-12.rtp.raleigh.ibm.com. So you need to configure the following DNS wildcard entries:

    *.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
    *.cps-rack-79-vm-12.xxx.xxx.xxx.xxx IN A x.xx.xx.15
    If you are unable to easily make changes to your DNS server, you can add the following entries to your local /etc/hosts file (or equivalent on Windows) for testing purposes. This will allow you to log in to the OpenShift console, but note that you would need additional entries for any applications you would deploy later.

    ocp_cluster_ip console-openshift-console.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx
    oauth-openshift.apps.cps-rack-79-vm-12.xxx.xxx.xxx.xxx
    You can find more information about OpenShift external DNS requirements here. The DNS records listed as “This record must be resolvable by both clients external to the cluster …” are required. DNS is also provided on the Helper Nodes to cover the resolution inside the cluster.

    If you are able to configure DNS records up front, then the cluster console link will be accessible immediately. Configuring DNS ahead of time is the recommended approach for deploying OpenShift Container Platform clusters on Cloud Pak System. You would need to create the following records in your DNS server for each IP in the IP group you are using to deploy (so that any IP that is selected from the IP group to be the floating IP for the cluster will already have wildcard entries associated with it in DNS):

    *.sub.domain IN A <ip>
    *.mycluster.sub.domain IN A <ip>

     

  5. Access your OpenShift cluster

    You can now access your OpenShift 4 cluster using the OpenShift console link as shown in Figure 30. 

    Access OCP console

    Figure 30: Access OCP console

    • Log in with the username kubeadmin and the password that you retrieved earlier.

     

    Login to OCP console

     Figure 31: Login to OCP console

    After you log in, you will see the console as shown in Figure 32.

     

    Log in to OCP console

     Figure 32: Log in to OCP console

    • Navigate to Compute > Nodes. If you see three Master nodes and five (or number of workers specified at time of deployment) worker nodes, then it confirms that the OpenShift with OCS cluster topology got deployed as expected.
  6. Verifying your OpenShift Container Platform with OCS cluster

    Inspect your cluster from IBM Cloud Pak Console > Manage accelerator instances

    A. Verify topology

    In the IBM Cloud Pak Console -> Manage accelerator instances page, click Nodes for the deployed instance as shown in Figure 33. You can verify cluster topology and VMs deployed as part of the instance.

     

    Cluster topology

     

    Figure 33: Cluster topology

    B. Verify middleware roles

    In the IBM Cloud Pak Console -> Manage accelerator instances, click Middleware for the deployed instance as shown in Figure 34. You can verify the cluster topology and deployed VMs as a part of the instance.

     

    Middleware view

     

    Figure 34: Middleware view

    NOTE: Do not stop or restart OpenShift Container Platform with OCS cluster or any nodes until after 24 hours of deployment. Failing to do so may render your cluster to a broken state that cannot be recovered. Refer this page.

  7. Registering your OpenShift Container Platform with OCS cluster with Red Hat

    Finally, do not forget to register your OpenShift Container Platform with OCS cluster with Red Hat. This manual step is required if your OpenShift cluster does not have internet access to reach Red Hat. You can follow step 4 here to register your cluster on the “Cluster registration” page.

  8. Next steps

    Now you are ready for workload deployment on your OpenShift Container Platform with OCS cluster where OCS is the persistent storage. Optionally, read this blog to understand how OCS cluster is created for OpenShift Container Platform in IBM Cloud Pak System.

  9. Conclusion

    IBM Cloud Pak System 2.3.3.3 enables clients to quickly roll out one or more Red Hat OpenShift Container Platform with OCS clusters, which greatly simplifies the process, ensures consistency, and avoids human error. This is also used as the foundation for the deployment of IBM Cloud Paks and other OCP workloads on the IBM Cloud Pak System platform.

Join The Discussion