Overview

Skill Level: Beginner

This article provide instructions of setting up datastage docker image 11.7.0.1 on Redhat Linux.

Step-by-step

  1. Provision 3 VMs of RHEL v7.4

    In my test environment, I got the following hosts after the provision, all with RHEL 7.4

    ds-docker1.fyre.ibm.com
    ds-docker2.fyre.ibm.com
    ds-docker3.fyre.ibm.com

  2. Install docker on each host

    Run the following command to install docker. You need to repeat this for each host.

    [root@ds-docker1 ~]#sudo yum install docker

  3. Configure the Password-less SSH login for the 3 hosts

    a.Generate the key

    [root@ds-docker1 ~]# ssh-keygen

    Generating public/private rsa key pair.

    Enter file in which to save the key (/root/.ssh/id_rsa):

    /root/.ssh/id_rsa already exists.

    Overwrite (y/n)? y

    Enter passphrase (empty for no passphrase):

    Enter same passphrase again:

    Your identification has been saved in /root/.ssh/id_rsa.

    Your public key has been saved in /root/.ssh/id_rsa.pub.

    The key fingerprint is:

    SHA256:uIbT6eokYAL+fOtUqcTyNxiMoTl/ydhN8V5o03VS0W4 root@ds-docker1.fyre.ibm.com

    The key’s randomart image is:

    +—[RSA 2048]—-+

    |               oo|

    |              . .|

    |.  .   .     o o |

    |o o =  .+ o . o E|

    |o* o =.+S= o   . |

    |o.= BoOoo o      |

    |  .=+@=+ .       |

    |   o++o .        |

    |   .++.          |

    +—-[SHA256]—–+

     

    b. Verify the public key is generated

    [root@ds-docker1 ~]# cat .ssh/id_rsa.pub

    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDL6Ob2z3Rkp0OhO40jcRw6lYzqE0Uk5ZMXlwVlZvrA6Jd2G/dst+U7T4nwhFBcdiykDbvop+TC6s4jMm48s/NCj2YJz6k6JXqjjwg+ewx0WsnvyKgO4x5ACrJmeT54n80+h42N+ygShRqoBqq81g885AQpYmjrVDY/GsQx72Jvwb/Qvs7badAbtpCvAlHgiM+7/5VSx/cR2tUWS9FM6cnlWVkeJV4WBGZ86IHp7YsIBj76V438BWhrRovW8nCqCp8uYhHCDW/m0KHV73o0ib6islBhC+5C4Zt4vza9qLSuJXERghkG0aqTmyw4CVsX8BxXeBjxFwBhOKvGzqY+dCIN root@ds-docker1.fyre.ibm.com

     

    c. Copy the public key to local host and remote hosts

    [root@ds-docker1 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@ds-docker2.fyre.ibm.com

    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: “.ssh/id_rsa.pub”

    The authenticity of host ‘ds-docker2.fyre.ibm.com (172.16.12.171)’ can’t be established.

    ECDSA key fingerprint is SHA256:WvjuiXJ+3XFlivNwL1wm+zNMLaRwGISlTNxhd76agD8.

    ECDSA key fingerprint is MD5:87:2d:03:2f:83:69:fc:f0:8c:6e:40:e0:19:f9:84:ba.

    Are you sure you want to continue connecting (yes/no)? yes

    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys

    root@ds-docker2.fyre.ibm.com’s password:

     

    Number of key(s) added: 1

     

    d. Now try logging into the machine and check to make sure that no password required for ssh login

    [root@ds-docker1 ~]# ssh root@ds-docker2.fyre.ibm.com

    Last login: Thu Nov 22 00:22:16 2018 from 172.16.12.160

     

    e. Repeat the same step on the 3rd host

    [root@ds-docker1 ~]# ssh-copy-id -i .ssh/id_rsa.pub root@ds-docker3.fyre.ibm.com

     

    f. Now verify that Password-less SSH login is properly configured between the 3 hosts

    [root@ds-docker1 ~]# ssh root@ds-docker2.fyre.ibm.com

    Last login: Thu Nov 22 00:22:16 2018 from 172.16.12.XXX

    [root@ds-docker2 ~]# ssh root@ds-docker3.fyre.ibm.com

    Last login: Thu Nov 22 00:22:25 2018 from 172.16.12.XXX

  4. Create installation directory of the datastage docker

    [root@ds-docker1 ~]# mkdir /ibm-ds-docker

  5. Extract the datastage docker image to the installation directory created in step 4

    [root@ds-docker1 ibm-ds-docker]# tar -zxvf ibm-ds-docker-inst-11.7.0.1-2.0.tar.gz
    DS-Kube-Installer/

    DS-Kube-Installer/license/

    DS-Kube-Installer/license/DataStage/

    DS-Kube-Installer/license/DataStage/License_Text/

    DS-Kube-Installer/license/DataStage/License_Text/LA_cs.txt

    DS-Kube-Installer/license/DataStage/License_Text/LA_de.txt

    DS-Kube-Installer/license/DataStage/License_Text/LA_el.txt

    …

    DS-Kube-Installer/ds-meter/manifests/

    DS-Kube-Installer/ds-meter/manifests/cluster-binding.yaml

    DS-Kube-Installer/ds-meter/manifests/ds-metering.yaml

    DS-Kube-Installer/ds-meter/manifests/metering-pv.yaml

    DS-Kube-Installer/ds-meter/manifests/metering-pvc.yaml

    DS-Kube-Installer/ds-meter/rpms/

    DS-Kube-Installer/ds-meter/rpms/ansible-2.4.1.0-1.el7.noarch.rpm

    DS-Kube-Installer/ds-meter/rpms/jq-1.3-2.el6.x86_64.rpm

    DS-Kube-Installer/deleteNodes.sh

    ibm-datastage-dockers.tar.gz

     

    [root@ds-docker1 ibm-ds-docker]# ls -ltr
    total 14652016

    -rw-r–r–¬† 1 root root 6588307956 Jul¬† 9 10:23 ibm-datastage-dockers.tar.gz

    -rw-r–r–¬† 1 root root 8415348711 Nov 22 18:26 ibm-ds-docker-inst-11.7.0.1-2.0.tar.gz

    drwxr-xr-x 14 root root       4096 Nov 22 18:31 DS-Kube-Installer

  6. Edit the datastageinfo.json

    Here is an example, be noted about the BOLD ones, those should match your own configuration

    [root@ds-docker1 DS-Kube-Installer]# cat datastageinfo.json

    {

    ¬† “__doc__Comments”: [

    ¬†¬†¬† “namespace —¬† A string value provided to Kubernetes¬† namespace. e.g. TEST DEV QA etc.”,

    ¬†¬†¬† “numberOfNodes — Total number of nodes used for the cluster during install.”,

    ¬†¬†¬† “distributedFileSystemType — Choice of distributed file-system (NFS or GlusterFS) default is nfs.”,

    ¬†¬†¬† “distributedDataDir — Mount directory on the distributed file-system server. The persistent volumes are created in this directory.”,

    ¬†¬†¬† “distributedFileServer — Node of choice for the distributed file-system server – default is the master-node.”,

    ¬†¬†¬† “enginePvSize — Persistent Volume size in Gigabytes for the Engine pod.”,

    ¬†¬†¬† “repositoryPvSize — Persistent Volume size in Gigabytes for the Repository pod.”,

    ¬†¬†¬† “servicesSPvize — Persistent Volume size in Gigabytes for the Services pod.”,

    ¬†¬†¬† “dockerRegistryPvSize — Docker registry persistent volume size in Gigabytes.”,

    ¬†¬†¬† “dockerRegistryPath — File System Path for docker registry images – default is relative to the Install directory.”,

    ¬†¬†¬† “masterNodeHost — hostname of node chosen to be the master node for Kubernetes install.”,

    ¬†¬†¬† “workerNodeHosts — List of worker nodes with labels if applicable for the kubernetes cluster install – add more to increase computes.”

      ],

    ¬† “licenseAccepted”: “no”,

    ¬† “namespace”: “test-wy“,

    ¬† “numberOfNodes”: “3“,

    ¬† “distributedFileSystemType”: “nfs”,

    ¬† “distributedDataDir”: “/data”,

    ¬† “distributedFileServer”: “ds-docker1.fyre.ibm.com“,

    ¬† “enginePvSize”: “100Gi”,

    ¬† “repositoryPvSize”: “100Gi”,

    ¬† “servicesPvSize”: “50Gi”,

    ¬† “dockerRegistryPvSize”: “50Gi”,

    ¬† “dockerRegistryPath”: “registry/data”,

    ¬† “isMeteringEnabled” : “no”,

    ¬† “masterNodeHost”: “ds-docker1.fyre.ibm.com“,

    ¬† “workerNodeHosts”: [

        {

    ¬†¬†¬†¬†¬† “label”: “is-repository”,

    ¬†¬†¬†¬†¬† “name”: “ds-docker2.fyre.ibm.com

        },

        {

    ¬†¬†¬†¬†¬† “label”: “is-engine”,

    ¬†¬†¬†¬†¬† “name”: “ds-docker3.fyre.ibm.com

        }

      ]

    }

  7. Deploy the datastage docker image

    The script may fail if your environment doesn’t meet the system requirement, ie. a cluster with at least 8GB of memory and 2 cores of CPU per node, and a minimum of 30 GB of disk space on the root file system for every node in the cluster.

     

    Also, the datastage docker image deployment also require the kernel version, you may need to update your kernel if you see the following error

    [root@ds-docker1 DS-Kube-Installer]# ./deployMultiNode.sh

    ****************************************************************

    Do you accept the license agreement as shown in

    <install-root>/license/DataStage/License_Text/Lic_en.txt? <YES|NO>

    ****************************************************************

    <YES|NO>: YES

    firewall_status=inactive

    [2018/11/21 11:40:37 PM] ###### System kernel version : 3.10.0

    [2018/11/21 11:40:37 PM] ##### ERROR: Failed minimum kernel level check!! Please ensure minimum kernel version is 4.1.52.el7.x86_64

     

    In this case, please see step 8 for update kernel. If you passed the pre-check, go to step 9

     

     

  8. Update kernel

    1. wget http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

    2. rpm -ivh elrepo-release-7.0-2.el7.elrepo.noarch.rpm

    3. Install the kernel-ml package from the is repo

    yum –enablerepo=elrepo-kernel install kernel-ml

    4. check grub2.cfg

    awk -F\’ /^menuentry/{print\$2} /etc/grub2.cfg

    [root@ds-docker1 DS-Kube-Installer]# awk -F\’ /^menuentry/{print\$2} /etc/grub2.cfg

    Red Hat Enterprise Linux Server (3.10.0-862.14.4.el7.x86_64) 7.4 (Maipo)

    Red Hat Enterprise Linux Server (3.10.0-693.21.1.el7.x86_64) 7.4 (Maipo)

    Red Hat Enterprise Linux Server (3.10.0-693.17.1.el7.x86_64) 7.4 (Maipo)

    Red Hat Enterprise Linux Server (0-rescue-f7bbe4af93974cbfa5c55b68c011d41c) 7.3 (Maipo)

    5. grub2-set-default 0

    Note: you could run these to check it:

    [root@ds-docker1 DS-Kube-Installer]# cat /boot/grub2/grubenv|grep saved

    saved_entry=0

    6. grub2-mkconfig -o /boot/grub2/grub.cfg

    [root@ds-docker1 DS-Kube-Installer]# grub2-mkconfig -o /boot/grub2/grub.cfg

    Generating grub configuration file …

    Found linux image: /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64

    Found initrd image: /boot/initramfs-3.10.0-862.14.4.el7.x86_64.img

    Found linux image: /boot/vmlinuz-3.10.0-693.21.1.el7.x86_64

    Found initrd image: /boot/initramfs-3.10.0-693.21.1.el7.x86_64.img

    Found linux image: /boot/vmlinuz-3.10.0-693.17.1.el7.x86_64

    Found initrd image: /boot/initramfs-3.10.0-693.17.1.el7.x86_64.img

    Found linux image: /boot/vmlinuz-0-rescue-f7bbe4af93974cbfa5c55b68c011d41c

    Found initrd image: /boot/initramfs-0-rescue-f7bbe4af93974cbfa5c55b68c011d41c.img

    done

    7. reboot

    8. Check the kernel version

    [root@ds-docker1 ~]# cat /proc/version

    Linux version 4.19.3-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)) #1 SMP Wed Nov 21 17:17:00 EST 2018

  9. Run deployMultiNode.sh

    Make sure you get something like the following

    [root@ds-docker1 DS-Kube-Installer]# ./deployMultiNode.sh

    PLAY RECAP ******************************************************************************************

    ds-docker1.fyre.ibm.com     : ok=36   changed=28   unreachable=0    failed=0

    ds-docker2.fyre.ibm.com     : ok=10   changed=4    unreachable=0    failed=0

    ds-docker3.fyre.ibm.com     : ok=10   changed=4    unreachable=0    failed=0

  10. Verify that the pods and services are in good status

    [root@ds-docker1 DS-Kube-Installer]# kubectl get pods -o wide -n test-wy

    NAME                                     READY     STATUS    RESTARTS   AGE       IP                N                                   ODE

    is-en-conductor-0                        1/1       Running   0          7m        192.168.123.193   d                                   sdocker3.fyre.ibm.com

    is-engine-compute-0                      1/1       Running   0          13m       192.168.123.192   d                                   sdocker3.fyre.ibm.com

    is-engine-compute-1                      1/1       Running   0          12m       192.168.31.193    d                                   sdocker2.fyre.ibm.com

    is-servicesdocker-pod-7d4778b854-vqss5   1/1       Running   0          10m       192.168.31.194    d                                   sdocker2.fyre.ibm.com

    is-xmetadocker-pod-5db6bc645-bzfxm       1/1       Running   0          13m       192.168.31.192    d

     

     

    [root@ds-docker1 DS-Kube-Installer]# kubectl get svc -n test-wy

    NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                           AGE

    conductor-0         ClusterIP   None            <none>        80/TCP                            20m

    en-cond             ClusterIP   None            <none>        80/TCP                            14m

    is-en-conductor-0   NodePort    10.98.249.248   <none>        31538:32501/TCP,31531:32502/TCP   14m

    is-servicesdocker   NodePort    10.100.5.143    <none>        9446:32500/TCP                    17m

    is-xmetadocker      NodePort    10.99.141.209   <none>        50000:31000/TCP                   20m

Join The Discussion