Red Hat announced the availability of OpenShift Container Platform on the IBM Z and IBM LinuxONE platforms last year, and since then we've noticed that many of our clients end up with a disconnected/restricted network installs, given the nature of critical business running on the platform.

This tutorial walks you through the process of a disconnected (a.k.a. restricted) network install. It assumes that you have the prerequisite infrastructure already in place (DNS, HAProxy, etc.) for all installations, and that you know how to proceed with a regular installation (generate manifests, artifacts, etc.) for s390x.

Note: These instructions work for versions 4.6/4.7/4.8. Be sure to replace the release version where it is appropriate.

Prerequisites

For this tutorial, you need a guest with access to the internet, either directly or through a proxy. You also need to install Podman and download the command-line utilities jq (for parsing JSON) and the OpenShift command-line interface (oc) to manage the cluster.

s390x or x86 Red Hat Enterprise Linux 7/8* (let's call this "utility-vm" from now on)
Podman
jq
oc client
utility-vm must have access to the internet, either directly or through a proxy

Note: With Red Hat Enterprise Linux 7, you have the option to use Docker or Podman, but for Red Hat Enterprise Linux 8 only Podman is available from the subscription repositories.

Estimated time

It should take you about 1 hour to complete this tutorial.

Steps

This tutorial is composed of 4 steps:

Create the private image registry for mirroring.
Merge the pull secrets.
Mirror the content.
Generate the install-config.yaml for the disconnected install.

flows

1. Install a private image registry

As this is a disconnected installation, OpenShift Container Platform, once operational, will not have access to the internet and needs to pull the required images from an existing location that does. For this, you need to create an image registry and mirror the images from the Red Hat registry. At the very least, utility-vm must have access to the internet, either directly or through a proxy. In the examples below, a proxy is used.

Here are the export proxy variables that will be used by Podman and later by oc commands:

$ export HTTP_PROXY=http://my-proxy.example.com:9090
$ export HTTPS_PROXY=http://my-proxy.example.com:9090
$ export NO_PROXY=.internal.example.com,.example.com

Note: This will almost certainly be different for you. Tweak as needed.

Install the registry:
```
$ yum install docker-distribution
```

Create the folders that will be used by the registry and the self-signed certificate:

$ mkdir -p /etc/docker-distribution/certs
$ cd /etc/docker-distribution/certs
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt

Create the folder that will be used by the registry (where the images will reside):
```
$ mkdir /var/registry/docker-distribution
```

Create a basic htpasswd authentication:

$ htpasswd -bBc /etc/docker-distribution/registry_passwd [user] [pass]

Edit the config file /etc/docker-distribution/registry/config.yml:

version: 0.1
log:
fields:
    service: registry
    environment: development
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/registry/docker-distribution
    delete:
        enabled: true
http:
    addr: :5000
    tls:
    certificate: /etc/docker-distribution/certs/domain.crt
    key: /etc/docker-distribution/certs/domain.key
    host: https://utility-vm.example.com:5000
    secret: testsecretrandom
    relativeurls: false
auth:
    htpasswd:
    realm: basic-realm
    path: /etc/docker-distribution/registry_passwd

Note: Replace the host and secret as needed.

Start the service:
```
$ systemctl start docker-distribution
```

Validate that the registry service is listening and working:

$ netstat -tulpn |grep 5000
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      55061/registry

$ curl -u openshift:redhat -k https://localhost:5000/v2/_catalog
{"repositories":[]}

Note: Your registry is still empty, so there are no repositories yet.

Trust the repositories:

$ cp /etc/docker-distribution/certs/domain.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust

Try to pull and push a sample image:

$ podman pull ubi7/ubi:7.7
Trying to pull registry.access.redhat.com/ubi7/ubi:7.7...
Getting image source signatures
Copying blob 1af260e02c36 done
Copying blob eb3a55bf2d3b done
Copying config f5c7f86b27 done
Writing manifest to image destination
Storing signatures
f5c7f86b27eb77d40f1e30109e1d35444b6bba9aaacc8d88eaa15f7617ca4d05

$ podman images |grep ubi7
registry.access.redhat.com/ubi7/ubi                                7.7                          f5c7f86b27eb   9 months ago    221 MB

$ podman tag registry.access.redhat.com/ubi7/ubi:7.7 utility-vm.example.com:5000/ubi7:7.7

$ podman images |grep ubi7
utility-vm.example.com:5000/ubi7                                    7.7                          f5c7f86b27eb   9 months ago    221 MB
registry.access.redhat.com/ubi7/ubi                                7.7                          f5c7f86b27eb   9 months ago    221 MB

$ podman push utility-vm.example.com:5000/ubi7:7.7
Getting image source signatures
Copying blob b51142454f75 done
Copying blob 9aaa11d10a00 done
Copying config f5c7f86b27 done
Writing manifest to image destination
Storing signatures

$ curl -u openshift:redhat -k https://utility-vm.example.com:5000/v2/_catalog
{"repositories":["ubi7"]}

Open the firewall ports:

$ firewall-cmd --permanent --add-port=5000/tcp
success

$ firewall-cmd --list-ports
53/udp

$ firewall-cmd --reload
success

$ firewall-cmd --list-ports
53/udp 5000/tcpv

2. Merge the pull secrets

An installation can only have a single pull secret. A pull secret is a JSON file that contains the keys that authorize you to pull images from the Red Hat registry. As the mirroring process pulls images from the Red Hat registry and your cluster pulls from your internal private registry, you need to merge both secrets into a single JSON file.

Create a pull secret for your local registry and output it to local_pullsecret.json:

$ podman login -u openshift -p redhat --authfile ./local_pullsecret.json utility-vm.example.com:5000

It should look like this:

{
    "auths": {
        "utility-vm.example.com:5000": {
            "auth": "c3BlNocaaWZ0OnJlZGhhdB=="
        }
    }
}

Get the pull secret from OpenShift Cluster Manager. If you have a corporate account, use your corporate credentials to login, otherwise, register a personal account with no charges.

Save it to a file named ocp_pullsecret.json. It should look like this:

{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOT...==","email":"johndoe@someemail.com"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...Ca==","email":"[your OpenShift Subscription Email]"}}}

Note: The real keys are much longer.

Merge the pull secrets using jq:

$ jq -c --argjson var "$(jq .auths ./local_pullsecret.json)" '.auths += $var' ./ocp_pullsecret.json > merged_pullsecret.json

Validate:
```
$ jq . merged_pullsecret.json
```

It should look similar to this -- notice your local pull secret is placed at the bottom of your merged_pullsecret.json file :

{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...==","email":"johndoe@someemail.com"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...==","email":"johndoe@someemail.com"},"utility-vm.example.com:5000":{"auth":"c3BlNocaaWZ0OnJlZGhhdB=="}}}

3. Mirror the content

Now that the pull secret is ready, you're all set to mirror the images from the Red Hat repo.

Set up the environment variables:

$ export OCP_RELEASE=4.6.9
$ export LOCAL_REGISTRY=utility-vm.example.com:5000
$ export LOCAL_REPOSITORY=ocp4/openshift4
$ export PRODUCT_REPO=openshift-release-dev
$ export LOCAL_SECRET_JSON=/var/registry/oc4.6/secrets/merged_pullsecret.json
$ export RELEASE_NAME=ocp-release
$ export ARCHITECTURE=s390x

Note 1: Ensure the release matches the exact output from the command openshift-install version that you downloaded.

Note 2: Set the merged pull secret path as per your environment.

Be sure to do a dry run before the real run.

$ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
    --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
    --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
    --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

Export GODEBUG=x509ignoreCN=0 to make oc bypass a Subject Alternative Name (SAN):
```
$ export GODEBUG=x509ignoreCN=0
```
Note: If you follow the self-signed certificate instructions described here, you should get the error below. To avoid this, you can create a certificate with SAN or use the variable described above. In the Resources, you can find a link on how to create a self-signed certificate with SAN.
```
error: unable to connect to utility-vm.example.com:5000/ocp4/openshift4: Get "https://utility-vm.example.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
```

Now you're ready to go for the real action and mirror the images:

$ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
    --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
    --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
    --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

You should see a lot of image names on the console -- at the bottom, you should see the important portion that you need to save. It should look similar to this:

info: Mirroring completed in 1m22.15s (61.22MB/s)

Success
Update image:  utility-vm.example.com:5000/ocp4/openshift4:4.6.9
Mirror prefix: utility-vm.example.com:5000/ocp4/openshift4

To use the new mirrored repository to install, add the following section to the install-config.yaml:

imageContentSources:
- mirrors:
  - utility-vm.example.com/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: example
spec:
repositoryDigestMirrors:
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

Note: Although you'll use only this portion of the output, I recommend saving the whole output including the image names. If things go wrong and you need to troubleshoot, it will be handy to have the entire list of images.

Validate that the new repository was created and you can pull an image:

$ curl -u openshift:redhat -k https://utility-vm.example.com:5000/v2/_catalog
{"repositories":["ocp4/openshift4","ubi7"]}

$ podman pull --authfile /var/registry/oc4.6/secrets/local_pullsecret.json utility-vm.example.com:5000/ocp4/openshift4:4.6.9-operator-lifecycle-manager
Trying to pull utility-vm.example.com:5000/ocp4/openshift4:4.6.9-operator-lifecycle-manager...
Getting image source signatures
Copying blob 84b951709aed skipped: already exists
Copying blob 69c1e0128b10 skipped: already exists
Copying blob 11606feade9d skipped: already exists
Copying blob 66f4d5600fff skipped: already exists
Copying blob bb78f045807f done
Copying config 4926ac11ab done
Writing manifest to image destination
Storing signatures
4926ac11abea54d9acffb3f8862bdb22b2aac11570c82f426038f2729f240421

Note: Replace the local_pullsecret.json path as per your environment.

4. Modify install-config.yaml

Now you'll need to generate the artifacts as per the regular instructions (see Red Hat OpenShift Installation Process Experiences on IBM Z/LinuxONE in Resources). This step focuses on the modifications required for install-config.yaml:

Add the ImageContentSourcePolicy that points to your new private image registry.
Add the certificate bundle to authenticate against your private image registry.

Extract openshift-install from the registry:
```
$ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
```
Note: As long as you have connectivity to the internet (using proxy or directly) and all environment variables are still set, the above command should work. If it doesn't, download openshift-install and make sure that you get the exact matching version! (The link for openshift-install requires a credential. If you have a corporate account, use your corporate credentials to log in; otherwise, register a personal account with no charges.)
Even though you extracted the contents from the registry or downloaded them from the Red Hat OpenShift Cluster Manager, make sure that you have the exact same version as the OCP_RELEASE variable; otherwise, you'll encounter errors during the bootstrap process where the temporary control plane won't be able to find the correct tags:
```
$ ./openshift-install version
./openshift-install 4.6.9
built from commit a48ad4a15b42102d1747d2f5f3b635deffb950b5
release image utility-vm.example.com:5000/ocp4/openshift4@sha256:3d77e9b0fd14a5c4d50995bbb17494a02f27a69f2ffa9771b29d112fe084699f
```

Your original install-config.yaml should look similar to this:

apiVersion: v1
baseDomain: example.com
compute:
- hyperthreading: Enabled
architecture: s390x
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
architecture: s390x
name: master
replicas: 3
metadata:
name: zcluster-dev
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
    hostPrefix: 23
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTBsc2t5aXJxZnBqYjBjZ25weWs6WldMQlExMzlNMDgzNoCaNoCaNoCaQ1hJNEpCWVhRWExOOEc0S0o3NEhFUFRNoCaNoCaNoCaWldBBQk1RSFBXxg==","email":"johndoe@someemail.com"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTBsc2t5aXJxZnBqYjBjZ25weWs6WldMQlExMzlNMDgzNoCaNoCaNoCaQ1hJNEpCWVhRWExOOEc0S0o3NEhFUFRNoCaNoCaNoCaWldBBQk1RSFBXxg==","email":"johndoe@someemail.com"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOTBMWjJZbWNxQng5RnBOR3kwRmtVelVIaTJTZG5Kdm1ZTXk4cTdMd2xaQjJ2NnllejNoCa1WmdfdHI5dnpCclZQbXc2bTAzZHdEM3h1X0Q1SHlvNzNoCa==","email":"johndoe@someemail.com"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOTBMWjJZbWNxQng5RnBOR3kwRmtVelVIaTJTZG5Kdm1ZTXk4cTdMd2xaQjJ2NnllejNoCa1WmdfdHI5dnpCclZQbXc2bTAzZHdEM3h1X0Q1SHlvNzNoCa==","email":"johndoe@someemail.com"},"utility-vm.example.com:5000":{"auth":"c3BlNocaaWZ0OnJlZGhhdB=="}}}'
sshKey: |
  ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm

Note 1: For further details on install-config.yaml, click here.

Note 2: This tutorial does not cover how to generate an ssh-key, which you'll need to update in install-config.yaml. This tutorial focuses on the requirements for disconnected install pieces only. For more details on the whole install process, check the Resources section in the right-hand column.

At the bottom, add the imageContentSources that you received as output after mirroring:

...
sshKey: |
  ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
imageContentSources:
- mirrors:
  - utility-vm.example.com/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

Add to the bottom the certificate bundle that you created in step 3 of Install a private image registry above:

...
sshKey: |
  ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
imageContentSources:
- mirrors:
  - utility-vm.example.com/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - utility-vm.example.com:5000/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIF9zCCA9+gAwIBAgIJAM3thbw0gvDfMA0GCSqGSIb3DQEBCwUAMIGRMQswCQYD
  VQQGEwJVUzEQMA4GA1UECAwHQXJpem9uYTEQMA4GA1UEBwwHUGhvZW5peDENMAsG
  ...<cutting some pieces off>...
  GvkXaraLH64hQC3+HqkHMAbzEPI/AS/u3CD5RugcwqtK7f1p+5vsgvRHuzWxCJaw
  8fBSNlPP2qQS1popWFNSIic2JexHVuYCuGVH1MkTW1C5zXmoFbEKXhpu/Q==
  -----END CERTIFICATE-----

Note: This is YAML, so it's very important to remember the two spaces tabulation. This is also true for the certificate bundle at the bottom -- you must tabulate each line properly as in the example above.

That's it for disconnected. From this point, you can follow the regular instructions. I'm referencing a few other tutorials where this is covered in detail.

Notes

This is a step-by-step walkthrough of the process. Certain steps can be skipped/optimized based on your environment. There are also ways to script this such as this set of scripts created and maintained by our colleague, Pat Fruth. We also have supported software like IBM Cloud Infrastructure Center which can automate this process as well as automate deployment of other z/VM- and KVM-based workloads on the IBM Z and LinuxONE platforms.

Summary

This tutorial has guided you through the process of creating the required environment for a disconnected OpenShift on Z install. It introduced the concept of a disconnected environment and showed you how to create your own image registry, mirror the required Red Hat images, and tweak install artifacts to consume the images from your own internal registry.

Now that you've completed this tutorial, you're ready to move forward with the OpenShift installation as detailed in the article Red Hat OpenShift Installation Process Experiences on IBM Z/LinuxONE.

See the Resources section of this tutorial (in the upper right-hand column) for more links on this topic.

Red Hat OpenShift restricted network (disconnected) installation on IBM Z

Red Hat OpenShift restricted network (disconnected) installation on IBM Z