Red Hat OpenShift restricted network (disconnected) installation on IBM Z
For nodes that don't have internet connectivity, you can mirror images from Red Hat repositories to an internal repository
By
Alexandre de Oliveira,
Elton de Souza
On this page
Red Hat announced the availability of OpenShift Container Platform on the IBM Z and IBM LinuxONE platforms last year, and since then we've noticed that many of our clients end up with a disconnected/restricted network installs, given the nature of critical business running on the platform.
This tutorial walks you through the process of a disconnected (a.k.a. restricted) network install. It assumes that you have the prerequisite infrastructure already in place (DNS, HAProxy, etc.) for all installations, and that you know how to proceed with a regular installation (generate manifests, artifacts, etc.) for s390x.
Note: These instructions work for versions 4.6/4.7/4.8. Be sure to replace the release version where it is appropriate.
Prerequisites
For this tutorial, you need a guest with access to the internet, either directly or through a proxy. You also need to install Podman and download the command-line utilities jq (for parsing JSON) and the OpenShift command-line interface (oc) to manage the cluster.
s390x or x86 Red Hat Enterprise Linux 7/8* (let's call this "utility-vm" from now on)
utility-vm must have access to the internet, either directly or through a proxy
Note: With Red Hat Enterprise Linux 7, you have the option to use Docker or Podman, but for Red Hat Enterprise Linux 8 only Podman is available from the subscription repositories.
Estimated time
It should take you about 1 hour to complete this tutorial.
As this is a disconnected installation, OpenShift Container Platform, once operational, will not have access to the internet and needs to pull the required images from an existing location that does. For this, you need to create an image registry and mirror the images from the Red Hat registry. At the very least, utility-vm must have access to the internet, either directly or through a proxy. In the examples below, a proxy is used.
Here are the export proxy variables that will be used by Podman and later by oc commands:
An installation can only have a single pull secret. A pull secret is a JSON file that contains the keys that authorize you to pull images from the Red Hat registry. As the mirroring process pulls images from the Red Hat registry and your cluster pulls from your internal private registry, you need to merge both secrets into a single JSON file.
Create a pull secret for your local registry and output it to local_pullsecret.json:
Get the pull secret from OpenShift Cluster Manager. If you have a corporate account, use your corporate credentials to login, otherwise, register a personal account with no charges.
Save it to a file named ocp_pullsecret.json. It should look like this:
Export GODEBUG=x509ignoreCN=0 to make oc bypass a Subject Alternative Name (SAN):
$ exportGODEBUG=x509ignoreCN=0
Note: If you follow the self-signed certificate instructions described here, you should get the error below. To avoid this, you can create a certificate with SAN or use the variable described above. In the Resources, you can find a link on how to create a self-signed certificate with SAN.
error: unable toconnectto utility-vm.example.com:5000/ocp4/openshift4: Get "https://utility-vm.example.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
Now you're ready to go for the real action and mirror the images:
You should see a lot of image names on the console -- at the bottom, you should see the important portion that you need to save. It should look similar to this:
info:Mirroring completed in 1m22.15s (61.22MB/s)Success
Update image: utility-vm.example.com:5000/ocp4/openshift4:4.6.9Mirror prefix:utility-vm.example.com:5000/ocp4/openshift4To use the new mirrored repository to install, add the following section to the install-config.yaml:
imageContentSources:-mirrors:-utility-vm.example.com/ocp4/openshift4source:quay.io/openshift-release-dev/ocp-release-mirrors:-utility-vm.example.com:5000/ocp4/openshift4source:quay.io/openshift-release-dev/ocp-v4.0-art-devTo use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:
apiVersion:operator.openshift.io/v1alpha1kind:ImageContentSourcePolicymetadata:name:examplespec:repositoryDigestMirrors:-mirrors:-utility-vm.example.com:5000/ocp4/openshift4source:quay.io/openshift-release-dev/ocp-release-mirrors:-utility-vm.example.com:5000/ocp4/openshift4source:quay.io/openshift-release-dev/ocp-v4.0-art-dev
Note: Although you'll use only this portion of the output, I recommend saving the whole output including the image names. If things go wrong and you need to troubleshoot, it will be handy to have the entire list of images.
Validate that the new repository was created and you can pull an image:
Add the ImageContentSourcePolicy that points to your new private image registry.
Add the certificate bundle to authenticate against your private image registry.
Extract openshift-install from the registry:
$ oc adm release extract -a ${LOCAL_SECRET_JSON}--command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
Note: As long as you have connectivity to the internet (using proxy or directly) and all environment variables are still set, the above command should work. If it doesn't, download openshift-install and make sure that you get the exact matching version! (The link for openshift-install requires a credential. If you have a corporate account, use your corporate credentials to log in; otherwise, register a personal account with no charges.)
Even though you extracted the contents from the registry or downloaded them from the Red Hat OpenShift Cluster Manager, make sure that you have the exact same version as the OCP_RELEASE variable; otherwise, you'll encounter errors during the bootstrap process where the temporary control plane won't be able to find the correct tags:
$ ./openshift-install version
./openshift-install 4.6.9
built from commit a48ad4a15b42102d1747d2f5f3b635deffb950b5
release image utility-vm.example.com:5000/ocp4/openshift4@sha256:3d77e9b0fd14a5c4d50995bbb17494a02f27a69f2ffa9771b29d112fe084699f
Your original install-config.yaml should look similar to this:
Note 1: For further details on install-config.yaml, click here.
Note 2: This tutorial does not cover how to generate an ssh-key, which you'll need to update in install-config.yaml. This tutorial focuses on the requirements for disconnected install pieces only. For more details on the whole install process, check the Resources section in the right-hand column.
At the bottom, add the imageContentSources that you received as output after mirroring:
Note: This is YAML, so it's very important to remember the two spaces tabulation. This is also true for the certificate bundle at the bottom -- you must tabulate each line properly as in the example above.
That's it for disconnected. From this point, you can follow the regular instructions. I'm referencing a few other tutorials where this is covered in detail.
Notes
This is a step-by-step walkthrough of the process. Certain steps can be skipped/optimized based on your environment. There are also ways to script this such as this set of scripts created and maintained by our colleague, Pat Fruth. We also have supported software like IBM Cloud Infrastructure Center which can automate this process as well as automate deployment of other z/VM- and KVM-based workloads on the IBM Z and LinuxONE platforms.
Summary
This tutorial has guided you through the process of creating the required environment for a disconnected OpenShift on Z install. It introduced the concept of a disconnected environment and showed you how to create your own image registry, mirror the required Red Hat images, and tweak install artifacts to consume the images from your own internal registry.
About cookies on this siteOur websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising.For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
