Introduction

Installing Red Hat OpenShift Container Platform using User-Provisioned Infrastructure (UPI) has the benefit of providing users with extreme flexibility. Because of this, the official OpenShift documentation is extensive and goes into great detail about how to configure many different options without mandating any specific products for a cluster. For example, UPI allows you to bring your own Domain Name System (DNS) server and load balancers, and it doesn't require a specific networking configuration. In other words, a UPI-based install allows you to build your cluster in almost any way. Combined with the network flexibility we have built into IBM Power Virtual Server Private Cloud, you can choose to configure the network for your cluster in the way that fits best into your needs and into your pre-existing architecture. So, while a UPI installation makes none of those decisions for you, newcomers to OpenShift are often overwhelmed and ask for a set of pre-validated steps.

This tutorial describes the requirements to install an OpenShift cluster on the Power Virtual Server Private Cloud infrastructure and also provides an example set of configurations and commands that will result in an OpenShift cluster on your Power Virtual Server Private Cloud infrastructure.

By the end of this tutorial, you should have a better understanding of:

What resources are required for installing an OpenShift cluster
How to configure a pre-selected set of resources on a Power Virtual Server Private Cloud infrastructure pod
How to tie these resources together using the OpenShift Installer to create an OpenShift Container Platform cluster

This tutorial does not include:

Identity and access management (IAM) set up in your IBM Cloud account to control access to your workspace and its components
Security concepts such as providing additional certificates
Post-installation configuration
Additional operators installation

Prerequisites

Before getting started, read through the installation documentation to become familiar with the concepts. For UPI on Power Virtual Server Private Cloud infrastructure, refer to the generic instructions for any platform at: https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/installing_on_any_platform/index.

If you happen to notice that there are installation instructions for IBM Power and Power Virtual Server, note that those are specific to bare-metal Power, and Installer-Provisioned-Infrastructure (IPI) for Power Virtual Server Public (also referred to as Power Virtual Server off-prem).

It is also recommended that you read the security documentation at: https://docs.openshift.com/container-platform/4.18/security/index.html (confirm your version), as well as other sections including Operators and other areas of concern matching your requirements.

Note that all OpenShift documentation links provided in this tutorial are for the 4.18 release. If you are installing another version of OpenShift, be sure to select the appropriate release version on the documentation site.

Estimated time

Due to the number of components being created, this entire installation procedure will take approximately five hours.

Creating cluster resources

This tutorial explains how to create the following resources manually:

A Power Virtual Server workspace: A virtual space that groups together your Power Virtual Server components, including your server instances, subnets, custom boot images, and virtual disks.
A Subnet: Within your workspace, you'll need to create a subnet. You can give it any CIDR you want. If you are planning to make this cluster accessible from one of your existing site network segments outside of your pod, you can work with IBM Support to configure connectivity at the Layer 3 level. For more information, contact your support representative.

The automation described in this tutorial enables you to create and configure the following resources:

An API load-balancer: All requests to your cluster go through a centralized API. Each control plane node in your cluster runs an API server pod. Incoming requests should be load-balanced across the nodes as a best practice. You should configure a load-balancer that exists outside your cluster to serve as the API load-balancer. The automation uses NGINX, an open-source web server.
Domain Name System (DNS): OpenShift relies on hostnames for everything from the nodes to the services running in the cluster. You are required to have a DNS server that you can use to add records to the master and worker nodes. This DNS provider must support wildcard DNS entries. For more details, refer: https://docs.openshift.com/container-platform/4.16/installing/installing_platform_agnostic/installing-platform-agnostic.html#installation-dns-user-infra_installing-platform-agnostic.
Domain Host Configuration Protocol (DHCP): CoreOS doesn't support cloud-init, which is how Power Virtual Server assigns static IP addresses programmatically. To use DHCP, you can simply set up our own DHCP server. You can use the IPs assigned by the Power Virtual Server to create a leases file for the server.
A bootstrap node: This is a temporary node that runs a minimal Kubernetes control plane and uses that to bootstrap the OpenShift control plane on the control plane nodes. It is deleted as soon as the control plane nodes have been configured.
Control plane and worker nodes: In traditional Power vernacular, these are logical partitions (LPARs) running a Linux distribution called Red Hat Enterprise Linux CoreOS (RHCOS). Control plane nodes will run the management software for the cluster. Worker nodes will run the application workloads.

OpenShift clusters on the Power Virtual Server Private Cloud must be created using UPI. This allows for maximum flexibility when creating cluster components, including what type of load balancer is used, how your network is configured, how you manage DNS records, and so on.

As we are not covering the topic of IAM, it is assumed that everything is created using the Default Resource Group under your IBM Cloud account. If you need to create more fine-grained access control, refer to the following link: https://cloud.ibm.com/iam/overview.

Steps

Perform the following steps to prepare an environment that will be used for your OpenShift cluster:

Install the IBM Cloud command-line interface (CLI) and the power-iaas plugin.
Create a Power Virtual Server workspace.
Create your subnet.
Create a helper node LPAR to host a DNS server, API load balancer, and DHCP server.
1. Create the LPAR.
2. Configure the helper node.
3. Run the helper node playbook.
4. Validate the binary packages.
5. Disable unused services.
Run the cluster create script for the private infrastructure.

Step 1. Install the IBM Cloud CLI

To install the IBM Cloud CLI (using the ibmcloud command) on your computer with the power-iaas (pi) plug-in if you have not already done so, refer to the following: Installing the stand-alone IBM Cloud CLI | IBM Cloud Docs

All the steps can be performed using the CLI or the IBM Cloud UI, but the examples in this tutorial use the CLI.

Step 2. Create a workspace

Run the following commands to create a workspace for your cluster.

$ ibmcloud login –apikey=<your_api_key> --no-region
$ ibmcloud pi workspace create –d <datacenter_name> openshift-workspace
$ ibmcloud pi workspace ls --private
$ ibmcloud pi workspace target <new_workspace_crn>

Here openshift-workspace is used as the workspace name.

Note: The power-iaas plug-in defaults to displaying help text that is specific to Power Virtual Server Public Cloud (off-prem). When you run the workspace target against a Power Virtual Server Private Cloud workspace, you can see the help text and CLI options that apply only to that workspace type.

Step 3. Create your subnet

The subnet creation is not included in the automated procedure as it will vary on a case-by-case basis. In addition, if you are not going to create a private subnet, you need to open a ticket with IBM Support to configure the subnet to allow communication between your workspace's subnet and the subnet from which you want your cluster to be accessible. In this tutorial, we are using a subnet in the workspace that is connected through L3 Border Gateway Protocol (BGP) to an internal employee network.

ibmcloud pi subnet create --peer-type L3BGP --cidr-block <CIDR> ocp-bgp-subnet

Here ocp-bgp-subnet is used as the subnet name.

Note: We're not specifying a DNS server here because we'll set up a helper node running DNS for this subnet later. You can then edit the subnet to use its IP as the DNS server.

Whether or not you configure the same type of subnet, the automation provided will be able to configure your workspace's subnet.

Work with the IBM Support team to configure communication between the new subnet and your existing subnet. For more information about Power Virtual Server networking use cases documentation, refer: https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-network_use_cases#bi-dir-ext-conn-bgp

Step 4. Create a helper node LPAR to host a DNS server, API load balancer, and DHCP server

Create the OpenShift Container Platform helper node.

This step creates a VM to run a few "helper" services for creating your cluster. Later in this tutorial, we will be using Red Hat's Helper Node project (https://github.com/redhat-cop/ocp4-helpernode) to configure it. (Note that there is no quick start document in that repository for Power Virtual Server environments.)

The playbook in that repository will install all the tools you need on the helper node, except for the IBM Cloud CLI. You'll need to install ibmcloud and the poweriaas plugin again as you did in step 2 above, but on the helper node this time
1. Copy an OS boot image from the image catalog into your workspace:
```
$ ibmcloud pi image create CentOS-Stream-9
```
  This operation should complete in approximately 5 minutes.
2. Create the VM.
```
$ ibmcloud pi instance create --image CentOS-Stream-9 --memory 4 --processor-type shared --processors 0.25 --storage-tier tier3 --subnets ocp-bgp-subnet --sys-type s1022 ocp-helper
```
  Here ocp-helper is used as the VM name. Use whatever system type is most appropriate for your environment. Here, we have chosen s1022.
Configure the helper node.
1. On the helper node, change the NetworkManager configuration to prevent cloudinit from owning the DNS by editing the /etc/NetworkManager/conf.d/99-cloud-init.conf file and removing dns=none.
2. Restart NetworkManager.
```
ocp-helper# systemctl restart NetworkManager
```
3. Add an authoritative nameserver in /etc/resolv.conf that will enable the helper node to resolve hostnames for package downloads. You can either manually change the nameserver field in the /etc/resolv.conf file or set it for the primary interface using the nmcli command.
  
  For example:
  
  Identify the name of your primary network interface using the nmcli command.
```
ocp-helper# nmcli con show
NAME         UUID                                  TYPE      DEVICE
System eth0  5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  eth0
System eth1  9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04  ethernet  eth1
lo           6f99bb7e-90aa-4352-8fbe-ff80e131c90b  loopback  lo
enp0s1       21e81df1-c71f-48dc-88a8-a07dd30db25b  ethernet  --
ocp-helper# nmcli con mod "System eth0" ipv4.dns <DNS_Server_IP>
```
4. Add the EPEL repo using the instructions at: https://docs.fedoraproject.org/en-US/epel/getting-started/#_centos_stream_9. At the time of this writing, the instructions were as follows:
```
ocp-helper# dnf config-manager --set-enabled crb && dnf install https://dl.fedoraproject.org/pub/epel/epel{,-next}-release-latest-9.noarch.rpm
```
5. Install Git, Ansible, and jq.
```
ocp-helper# dnf install git ansible jq systemd-resolved
```
6. Install yq from: https://github.com/mikefarah/yq/releases/download/v4.35.2/yq_linux_ppc64le (check for a later release)
```
curl -fsSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.35.2/yq_linux_ppc64le && chmod +x /usr/local/bin/yq
```
  Note: yq is a small utility much like jq that works with YAML.
Run the helper node playbook.
Verify that the following binary files are the correct ones for ppc64le.
1. Validate the binary packages by making sure that the following commands run properly:
```
ocp-helper# oc version
ocp-helper# openshift-install version
ocp-helper# helm version
```
2. If errors are displayed, correct the vars.yaml values and run the following command:
```
ocp-helper# ansible-playbook -e @vars.yaml tasks/main.yml -e force_ocp_download=true
```
Disable unused services.

The helper node scripts for Power will install and configure a Trivial File Transfer Protocol (TFTP) server. For the cluster installation procedure we are using, we have imported the Red Hat Cores OS image directly into your workspace. Disable the TFTP service:

   ocp-helper# systemctl stop tftp.socket
   ocp-helper# systemctl disable tftp.socket
   ocp-helper# systemctl stop tftp.service
   ocp-helper# systemctl disable tftp.service

Step 5. Run the cluster create script to create a cluster for your private infrastructure

Now that you have set up the helper node, you can create your OpenShift cluster. We have provided a cluster creation script that can be found in the following GitHub repo: https://github.com/ocp-power-automation/ocp4-extras

The script and instructions for its usage are included in the README.md file in directory with the script at: https://github.com/ocp-power-automation/ocp4-extras/blob/main/powervs-private-scripts

Post-installation operations

It is recommended but not required that you configure an external storage to use with the OpenShift cluster. This tutorial does not guide you through configuring a specific storage solution. These are the two primary use cases for storage in OpenShift that you may want to configure:

Registry storage: OpenShift clusters contain an internal image registry. For production clusters, you should set up an external storage space on which the cluster can store container images. By having the storage external to the nodes themselves, you are protected against data loss in the case of node failure.
Application storage: In addition to storage for the container registry, some applications may require their own storage volumes. Containerized workloads rely on the Kubernetes concept of Volumes and Persistent Volumes. Volumes managed by the cluster or an admin can be shared with a Kubernetes Pod regardless of the node on which it is deployed.

Further reading

Summary

After you have successfully created a cluster, you would have a better understanding of the components required for OpenShift on the Power Virtual Server Private Cloud infrastructure. As stated in the introduction, there are countless ways in which you may want to configure your cluster. If you have existing software and services you would prefer to integrate, you can use the sample script provided as a template for how to do so. In addition, IBM provides many offerings that integrate OpenShift into Power Virtual Server. Reach out to IBM for further information on solutions available to you.

Install Red Hat OpenShift on IBM Power Virtual Server Private Cloud infrastructure

Install Red Hat OpenShift on IBM Power Virtual Server Private Cloud infrastructure