Win $20,000. Help build the future of education. Answer the call. Learn more

Red Hat OpenShift restricted network (disconnected) installation on IBM Z

Red Hat announced the availability of OpenShift Container Platform on the IBM Z and IBM LinuxONE platforms last year, and since then we’ve noticed that many of our clients end up with a disconnected/restricted network installs, given the nature of critical business running on the platform.

This tutorial walks you through the process of a disconnected (a.k.a. restricted) network install. It assumes that you have the prerequisite infrastructure already in place (DNS, HAProxy, etc.) for all installations, and that you know how to proceed with a regular installation (generate manifests, artifacts, etc.) for s390x.

Note: These instructions work for versions 4.6/4.7/4.8. Be sure to replace the release version where it is appropriate.

Prerequisites

For this tutorial, you need a guest with access to the internet, either directly or through a proxy. You also need to install Podman and download the command-line utilities jq (for parsing JSON) and the OpenShift command-line interface (oc) to manage the cluster.

  • s390x or x86 Red Hat Enterprise Linux 7/8* (let’s call this “utility-vm” from now on)
  • Podman
  • jq
  • oc client
  • utility-vm must have access to the internet, either directly or through a proxy

Note: With Red Hat Enterprise Linux 7, you have the option to use Docker or Podman, but for Red Hat Enterprise Linux 8 only Podman is available from the subscription repositories.

Estimated time

It should take you about 1 hour to complete this tutorial.

Steps

This tutorial is composed of 4 steps:

  1. Create the private image registry for mirroring.
  2. Merge the pull secrets.
  3. Mirror the content.
  4. Generate the install-config.yaml for the disconnected install.

flows

1. Install a private image registry

As this is a disconnected installation, OpenShift Container Platform, once operational, will not have access to the internet and needs to pull the required images from an existing location that does. For this, you need to create an image registry and mirror the images from the Red Hat registry. At the very least, utility-vm must have access to the internet, either directly or through a proxy. In the examples below, a proxy is used.

  1. Here are the export proxy variables that will be used by Podman and later by oc commands:

    $ export HTTP_PROXY=http://my-proxy.example.com:9090
    $ export HTTPS_PROXY=http://my-proxy.example.com:9090
    $ export NO_PROXY=.internal.example.com,.example.com
    

    Note: This will almost certainly be different for you. Tweak as needed.

  2. Install the registry:

    $ yum install docker-distribution
    
  3. Create the folders that will be used by the registry and the self-signed certificate:

    $ mkdir -p /etc/docker-distribution/certs
    $ cd /etc/docker-distribution/certs
    $ openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt
    
  4. Create the folder that will be used by the registry (where the images will reside):

    $ mkdir /var/registry/docker-distribution
    
  5. Create a basic htpasswd authentication:

    $ htpasswd -bBc /etc/docker-distribution/registry_passwd [user] [pass]
    
  6. Edit the config file /etc/docker-distribution/registry/config.yml:

    version: 0.1
    log:
    fields:
        service: registry
        environment: development
    storage:
        cache:
            layerinfo: inmemory
        filesystem:
            rootdirectory: /var/registry/docker-distribution
        delete:
            enabled: true
    http:
        addr: :5000
        tls:
        certificate: /etc/docker-distribution/certs/domain.crt
        key: /etc/docker-distribution/certs/domain.key
        host: https://utility-vm.example.com:5000
        secret: testsecretrandom
        relativeurls: false
    auth:
        htpasswd:
        realm: basic-realm
        path: /etc/docker-distribution/registry_passwd
    

    Note: Replace the host and secret as needed.

  7. Start the service:

    $ systemctl start docker-distribution
    
  8. Validate that the registry service is listening and working:

    $ netstat -tulpn |grep 5000
    tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      55061/registry
    
    $ curl -u openshift:redhat -k https://localhost:5000/v2/_catalog
    {"repositories":[]}
    

    Note: Your registry is still empty, so there are no repositories yet.

  9. Trust the repositories:

    $ cp /etc/docker-distribution/certs/domain.crt /etc/pki/ca-trust/source/anchors/
    update-ca-trust
    
  10. Try to pull and push a sample image:

    $ podman pull ubi7/ubi:7.7
    Trying to pull registry.access.redhat.com/ubi7/ubi:7.7...
    Getting image source signatures
    Copying blob 1af260e02c36 done  
    Copying blob eb3a55bf2d3b done  
    Copying config f5c7f86b27 done  
    Writing manifest to image destination
    Storing signatures
    f5c7f86b27eb77d40f1e30109e1d35444b6bba9aaacc8d88eaa15f7617ca4d05
    
    $ podman images |grep ubi7
    registry.access.redhat.com/ubi7/ubi                                7.7                          f5c7f86b27eb   9 months ago    221 MB
    
    $ podman tag registry.access.redhat.com/ubi7/ubi:7.7 utility-vm.example.com:5000/ubi7:7.7
    
    $ podman images |grep ubi7
    utility-vm.example.com:5000/ubi7                                    7.7                          f5c7f86b27eb   9 months ago    221 MB
    registry.access.redhat.com/ubi7/ubi                                7.7                          f5c7f86b27eb   9 months ago    221 MB
    
    $ podman push utility-vm.example.com:5000/ubi7:7.7
    Getting image source signatures
    Copying blob b51142454f75 done  
    Copying blob 9aaa11d10a00 done  
    Copying config f5c7f86b27 done  
    Writing manifest to image destination
    Storing signatures
    
    $ curl -u openshift:redhat -k https://utility-vm.example.com:5000/v2/_catalog
    {"repositories":["ubi7"]}
    
  11. Open the firewall ports:

    $ firewall-cmd --permanent --add-port=5000/tcp
    success
    
    $ firewall-cmd --list-ports
    53/udp
    
    $ firewall-cmd --reload
    success
    
    $ firewall-cmd --list-ports
    53/udp 5000/tcpv
    

2. Merge the pull secrets

An installation can only have a single pull secret. A pull secret is a JSON file that contains the keys that authorize you to pull images from the Red Hat registry. As the mirroring process pulls images from the Red Hat registry and your cluster pulls from your internal private registry, you need to merge both secrets into a single JSON file.

  1. Create a pull secret for your local registry and output it to local_pullsecret.json:

    $ podman login -u openshift -p redhat --authfile ./local_pullsecret.json utility-vm.example.com:5000
    
  2. It should look like this:

    {
        "auths": {
            "utility-vm.example.com:5000": {
                "auth": "c3BlNocaaWZ0OnJlZGhhdB=="
            }
        }
    }
    
  3. Get the pull secret from OpenShift Cluster Manager. If you have a corporate account, use your corporate credentials to login, otherwise, register a personal account with no charges.

    Save it to a file named ocp_pullsecret.json. It should look like this:

    {"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOT...==","email":"johndoe@someemail.com"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...Ca==","email":"[your Open Shift Subscription Email]"}}}
    

    Note: The real keys are much longer.

  4. Merge the pull secrets using jq:

    $ jq -c --argjson var "$(jq .auths ./local_pullsecret.json)" '.auths += $var' ./ocp_pullsecret.json > merged_pullsecret.json
    
  5. Validate:

    $ jq . merged_pullsecret.json
    
  6. It should look similar to this — notice your local pull secret is placed at the bottom of your merged_pullsecret.json file :

    {"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTB...==","email":"johndoe@someemail.com"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...==","email":"johndoe@someemail.com"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU...==","email":"johndoe@someemail.com"},"utility-vm.example.com:5000":{"auth":"c3BlNocaaWZ0OnJlZGhhdB=="}}}
    

3. Mirror the content

Now that the pull secret is ready, you’re all set to mirror the images from the Red Hat repo.

  1. Set up the environment variables:

    $ export OCP_RELEASE=4.6.9
    $ export LOCAL_REGISTRY=utility-vm.example.com:5000
    $ export LOCAL_REPOSITORY=ocp4/openshift4
    $ export PRODUCT_REPO=openshift-release-dev 
    $ export LOCAL_SECRET_JSON=/var/registry/oc4.6/secrets/merged_pullsecret.json 
    $ export RELEASE_NAME=ocp-release 
    $ export ARCHITECTURE=s390x
    

    Note 1: Ensure the release matches the exact output from the command openshift-install version that you downloaded.

    Note 2: Set the merged pull secret path as per your environment.

  2. Be sure to do a dry run before the real run.

    $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
        --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
        --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
        --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run
    
  3. Export GODEBUG=x509ignoreCN=0 to make oc bypass a Subject Alternative Name (SAN):

    $ export GODEBUG=x509ignoreCN=0
    

    Note: If you follow the self-signed certificate instructions described here, you should get the error below. To avoid this, you can create a certificate with SAN or use the variable described above. In the Resources, you can find a link on how to create a self-signed certificate with SAN.

    error: unable to connect to utility-vm.example.com:5000/ocp4/openshift4: Get "https://utility-vm.example.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
    
  4. Now you’re ready to go for the real action and mirror the images:

    $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
        --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
        --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
        --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}
    
  5. You should see a lot of image names on the console — at the bottom, you should see the important portion that you need to save. It should look similar to this:

    info: Mirroring completed in 1m22.15s (61.22MB/s)
    
    Success
    Update image:  utility-vm.example.com:5000/ocp4/openshift4:4.6.9
    Mirror prefix: utility-vm.example.com:5000/ocp4/openshift4
    
    To use the new mirrored repository to install, add the following section to the install-config.yaml:
    
    imageContentSources:
    - mirrors:
      - utility-vm.example.com/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-release
    - mirrors:
      - utility-vm.example.com:5000/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    
    To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:
    
    apiVersion: operator.openshift.io/v1alpha1
    kind: ImageContentSourcePolicy
    metadata:
    name: example
    spec:
    repositoryDigestMirrors:
    - mirrors:
      - utility-vm.example.com:5000/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-release
    - mirrors:
      - utility-vm.example.com:5000/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    

    Note: Although you’ll use only this portion of the output, I recommend saving the whole output including the image names. If things go wrong and you need to troubleshoot, it will be handy to have the entire list of images.

  6. Validate that the new repository was created and you can pull an image:

    $ curl -u openshift:redhat -k https://utility-vm.example.com:5000/v2/_catalog
    {"repositories":["ocp4/openshift4","ubi7"]}
    
    $ podman pull --authfile /var/registry/oc4.6/secrets/local_pullsecret.json utility-vm.example.com:5000/ocp4/openshift4:4.6.9-operator-lifecycle-manager
    Trying to pull utility-vm.example.com:5000/ocp4/openshift4:4.6.9-operator-lifecycle-manager...
    Getting image source signatures
    Copying blob 84b951709aed skipped: already exists  
    Copying blob 69c1e0128b10 skipped: already exists  
    Copying blob 11606feade9d skipped: already exists  
    Copying blob 66f4d5600fff skipped: already exists  
    Copying blob bb78f045807f done  
    Copying config 4926ac11ab done  
    Writing manifest to image destination
    Storing signatures
    4926ac11abea54d9acffb3f8862bdb22b2aac11570c82f426038f2729f240421
    

    Note: Replace the local_pullsecret.json path as per your environment.

4. Modify install-config.yaml

Now you’ll need to generate the artifacts as per the regular instructions (see Red Hat OpenShift Installation Process Experiences on IBM Z/LinuxONE in Resources). This step focuses on the modifications required for install-config.yaml:

  • Add the ImageContentSourcePolicy that points to your new private image registry.
  • Add the certificate bundle to authenticate against your private image registry.
  1. Extract openshift-install from the registry:

    $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
    

    Note: As long as you have connectivity to the internet (using proxy or directly) and all environment variables are still set, the above command should work. If it doesn’t, download openshift-install and make sure that you get the exact matching version! (The link for openshift-install requires a credential. If you have a corporate account, use your corporate credentials to log in; otherwise, register a personal account with no charges.)

  2. Even though you extracted the contents from the registry or downloaded them from the Red Hat OpenShift Cluster Manager, make sure that you have the exact same version as the OCP_RELEASE variable; otherwise, you’ll encounter errors during the bootstrap process where the temporary control plane won’t be able to find the correct tags:

    $ ./openshift-install version
    ./openshift-install 4.6.9
    built from commit a48ad4a15b42102d1747d2f5f3b635deffb950b5
    release image utility-vm.example.com:5000/ocp4/openshift4@sha256:3d77e9b0fd14a5c4d50995bbb17494a02f27a69f2ffa9771b29d112fe084699f
    
  3. Your original install-config.yaml should look similar to this:

    apiVersion: v1
    baseDomain: example.com
    compute:
    - hyperthreading: Enabled   
    architecture: s390x
    name: worker
    replicas: 0 
    controlPlane:
    hyperthreading: Enabled   
    architecture: s390x
    name: master 
    replicas: 3 
    metadata:
    name: zcluster-dev 
    networking:
    clusterNetwork:
    - cidr: 10.128.0.0/14 
        hostPrefix: 23 
    networkType: OpenShiftSDN
    serviceNetwork: 
    - 172.30.0.0/16
    platform:
    none: {}
    fips: false
    pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTBsc2t5aXJxZnBqYjBjZ25weWs6WldMQlExMzlNMDgzNoCaNoCaNoCaQ1hJNEpCWVhRWExOOEc0S0o3NEhFUFRNoCaNoCaNoCaWldBBQk1RSFBXxg==","email":"johndoe@someemail.com"},"quay.io":{"auth":"c8BlbnNoaWZNoCaNoCaNoCatZGV2K2FsZXhvbGlfcmgxdThwbGY2cTBsc2t5aXJxZnBqYjBjZ25weWs6WldMQlExMzlNMDgzNoCaNoCaNoCaQ1hJNEpCWVhRWExOOEc0S0o3NEhFUFRNoCaNoCaNoCaWldBBQk1RSFBXxg==","email":"johndoe@someemail.com"},"registry.connect.redhat.com":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOTBMWjJZbWNxQng5RnBOR3kwRmtVelVIaTJTZG5Kdm1ZTXk4cTdMd2xaQjJ2NnllejNoCa1WmdfdHI5dnpCclZQbXc2bTAzZHdEM3h1X0Q1SHlvNzNoCa==","email":"johndoe@someemail.com"},"registry.redhat.io":{"auth":"aTI1MT...ViXzVFbEwzclI4UHE1TTZYenpoWTBzSU45UlliejBNeW90WnBFVE1sdUZkTUFhOTBMWjJZbWNxQng5RnBOR3kwRmtVelVIaTJTZG5Kdm1ZTXk4cTdMd2xaQjJ2NnllejNoCa1WmdfdHI5dnpCclZQbXc2bTAzZHdEM3h1X0Q1SHlvNzNoCa==","email":"johndoe@someemail.com"},"utility-vm.example.com:5000":{"auth":"c3BlNocaaWZ0OnJlZGhhdB=="}}}'
    sshKey: |
      ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
    

    Note 1: For further details on install-config.yaml, click here.

    Note 2: This tutorial does not cover how to generate an ssh-key, which you’ll need to update in install-config.yaml. This tutorial focuses on the requirements for disconnected install pieces only. For more details on the whole install process, check the Resources section in the right-hand column.

  4. At the bottom, add the imageContentSources that you received as output after mirroring:

    ...
    sshKey: |
      ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
    imageContentSources:
    - mirrors:
      - utility-vm.example.com/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-release
    - mirrors:
      - utility-vm.example.com:5000/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    
  5. Add to the bottom the certificate bundle that you created in step 3 of Install a private image registry above:

    ...
    sshKey: |
      ssh-rsa AAANacoNacoNaco....NacoNacoNaco== root@utility-vm
    imageContentSources:
    - mirrors:
      - utility-vm.example.com/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-release
    - mirrors:
      - utility-vm.example.com:5000/ocp4/openshift4
      source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    additionalTrustBundle: |
      -----BEGIN CERTIFICATE-----
      MIIF9zCCA9+gAwIBAgIJAM3thbw0gvDfMA0GCSqGSIb3DQEBCwUAMIGRMQswCQYD
      VQQGEwJVUzEQMA4GA1UECAwHQXJpem9uYTEQMA4GA1UEBwwHUGhvZW5peDENMAsG
      ...<cutting some pieces off>...
      GvkXaraLH64hQC3+HqkHMAbzEPI/AS/u3CD5RugcwqtK7f1p+5vsgvRHuzWxCJaw
      8fBSNlPP2qQS1popWFNSIic2JexHVuYCuGVH1MkTW1C5zXmoFbEKXhpu/Q==
      -----END CERTIFICATE-----
    

    Note: This is YAML, so it’s very important to remember the two spaces tabulation. This is also true for the certificate bundle at the bottom — you must tabulate each line properly as in the example above.

  6. That’s it for disconnected. From this point, you can follow the regular instructions. I’m referencing a few other tutorials where this is covered in detail.

Notes

  1. This is a step-by-step walkthrough of the process. Certain steps can be skipped/optimized based on your environment. There are also ways to script this such as this set of scripts created and maintained by our colleague, Pat Fruth. We also have supported software like IBM Cloud Infrastructure Center which can automate this process as well as automate deployment of other z/VM- and KVM-based workloads on the IBM Z and LinuxONE platforms.

Summary

This tutorial has guided you through the process of creating the required environment for a disconnected OpenShift on Z install. It introduced the concept of a disconnected environment and showed you how to create your own image registry, mirror the required Red Hat images, and tweak install artifacts to consume the images from your own internal registry.

Now that you’ve completed this tutorial, you’re ready to move forward with the OpenShift installation as detailed in the article Red Hat OpenShift Installation Process Experiences on IBM Z/LinuxONE.

See the Resources section of this tutorial (in the upper right-hand column) for more links on this topic.