This tutorial features detailed step-by-step instructions for building a fully automated Continuous Integration/Continuous Deployment (CI/CD) pipeline for your IBM® z/OS® testing activities. It provides extensive details on an end-to-end automation proof-of-concept (POC) project, along with ample guidance and ideas to help you develop a concept for successfully building your CI/CD pipeline to drive fully automated z/OS testing capabilities.

Prerequisites

To complete this tutorial, you’ll need:

Tools/components:

REXX EXECs and JCL jobs
A GitHub repository
Jenkins software product
IBM z/OS Connector Jenkins Plugin
A simple Windows Batch Script (.bat file)
Jenkinsfile

Skills/experience: It helps to have some preliminary skills and working experience with the tools/components listed above. Knowing at least how they work is desirable; however, this is not mandatory, as the tutorial provides ample details to help you get up and running with a z/OS automation POC project.

Version-level requirements: It is always desirable and recommended to use the latest, supported version and maintenance levels with open source tools/plugins such as GitHub, Jenkins, and IBM z/OS Connector Jenkins Plugin. You can always download the latest, supported version and maintenance levels using the links provided in the Tools/components list above.

Any IBM Z® hardware and z/OS version should work just fine. Also, it is assumed that the POC project will be undertaken on a system that runs the Microsoft Windows Operating System.

Estimated time

Completing this tutorial should take about 3 to 4 hours.

1

Code some REXX EXECs

First, you need to code four simple REXX EXECs. Let's create the following REXX EXECs:

SMPREX00
SMPREX01
SMPREX02
SMPREX03

SMPREX00

****** ********************************* Top of Data **********************************
000001 /* REXX **************************************************************/         
000002 SAY ' >>>>> '                                                                   
000003 SAY '  REXX PROGRAM #01'                                                        
000004 SAY ' <<<<< '                                                                   
****** ******************************** Bottom of Data ********************************

SMPREX01

********************************* Top of Data **********************************
/* REXX **************************************************************/         
SAY ' >>>>> '                                                                   
SAY '  REXX PROGRAM #02'                                                        
SAY ' <<<<< '                                                                   
******************************** Bottom of Data ********************************

SMPREX02

********************************* Top of Data **********************************
/* REXX **************************************************************/         
SAY ' >>>>> '                                                                   
SAY '  REXX PROGRAM #03'                                                        
SAY ' <<<<< '                                                                   
******************************** Bottom of Data ********************************

SMPREX03

********************************* Top of Data **********************************
/* REXX **************************************************************/         
SAY ' >>>>> '                                                                   
SAY '  REXX PROGRAM #04'                                                        
SAY ' <<<<< '                                                                   
******************************** Bottom of Data ********************************

As you can see, each REXX Program only prints out three display messages. This is to keep things simple -- remember, you are not here to learn how to code REXX EXECs!

2

Code some JCL jobs

Now it's time to code four JCL jobs. Let's name them REXEXC00, REXEXC01, REXEXC02, and REXEXC03.

Job REXEXC00 executes the first REXX Program, SMPREX00, in the first job step, and sets the maximum condition code of the job to 01 in the second and final job step:

REXEXC00

********************************* Top of Data **********************************
//REXEXC00 JOB (5990),'BATCH IVP',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,            
//      TIME=NOLIMIT                                                            
//**********************************************************************        
//**********************************************************************        
//REXJ     EXEC PGM=IKJEFT01,DYNAMNBR=45                                        
//SYSPROC  DD DISP=SHR,DSN=HLQ1.HLQ2.JOBS                                  
//SYSTSPRT DD SYSOUT=*                                                          
//SYSTSIN  DD *                                                                 
%SMPREX00                                                                       
//*                                                                             
//IDCAMS   EXEC PGM=IDCAMS,COND=(0,NE,REXJ)                                     
//SYSPRINT DD SYSOUT=*                                                          
//SYSOUT   DD SYSOUT=*                                                          
//SYSIN    DD *                                                                 
 SET MAXCC=1                                                                    
//*                                                                             
//                                                                              
******************************** Bottom of Data ********************************

Note: DDName SYSPROC references the partitioned dataset (PDS) that houses the REXX EXEC/Program SMPREX00 source code as a member.

Job REXEXC01 executes the REXX program, SMPREX01:

REXEXC01

********************************* Top of Data **********************************
//REXEXC01 JOB (5990),'BATCH IVP',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,            
//      TIME=NOLIMIT                                                            
//**********************************************************************        
//**********************************************************************        
//REXJ     EXEC PGM=IKJEFT01,DYNAMNBR=45                                        
//SYSPROC  DD DISP=SHR,DSN=HLQ1.HLQ2.JOBS                                  
//SYSTSPRT DD SYSOUT=*                                                          
//SYSTSIN  DD *                                                                 
%SMPREX01                                                                       
//*                                                                             
//                                                                              
******************************** Bottom of Data ********************************

Job REXEXC02 executes the third REXX Program, SMPREX02, in the first job step, and sets the maximum condition code of the job to 02 in the second and final job step:

REXEXC02

********************************* Top of Data **********************************
//REXEXC02 JOB (5990),'BATCH IVP',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,            
//      TIME=NOLIMIT                                                            
//**********************************************************************        
//**********************************************************************        
//REXJ     EXEC PGM=IKJEFT01,DYNAMNBR=45                                        
//SYSPROC  DD DISP=SHR,DSN=HLQ1.HLQ2.JOBS                                  
//SYSTSPRT DD SYSOUT=*                                                          
//SYSTSIN  DD *                                                                 
%SMPREX02                                                                       
//*                                                                             
//IDCAMS   EXEC PGM=IDCAMS,COND=(0,NE,REXJ)                                     
//SYSPRINT DD SYSOUT=*                                                          
//SYSOUT   DD SYSOUT=*                                                          
//SYSIN    DD *                                                                 
 SET MAXCC=2                                                                    
//*                                                                             
//                                                                              
******************************** Bottom of Data ********************************

Job REXEXC03 executes the fourth REXX Program, SMPREX03:

REXEXC03

********************************* Top of Data **********************************
//REXEXC03 JOB (5990),'BATCH IVP',CLASS=A,MSGCLASS=X,NOTIFY=&SYSUID,            
//      TIME=NOLIMIT                                                            
//**********************************************************************        
//**********************************************************************        
//REXJ     EXEC PGM=IKJEFT01,DYNAMNBR=45                                        
//SYSPROC  DD DISP=SHR,DSN=HLQ1.HLQ2.JOBS                                  
//SYSTSPRT DD SYSOUT=*                                                          
//SYSTSIN  DD *                                                                 
%SMPREX03                                                                       
//*                                                                             
//                                                                              
******************************** Bottom of Data ********************************

3

Create your first "freestyle project" item in Jenkins

Next, you'll spend some time playing around with the Jenkins software. You will soon create your first "freestyle project" item in Jenkins, but before that you need to make sure that you've set up a few things correctly.

You should set up a GitHub repository for the project. Once you have created a repo for the project, you should clone it to your local machine. (I used my laptop configured with the Microsoft Windows Operating System.)

Go to your repo, click on the Code button, and copy the web URL

Code button

Repo -- Code button

Open the Windows command prompt and use the git clone command to copy the repo to your local machine. In my case, I copied my repo to my Windows desktop. Before executing the git clone command, I had to make sure that I had an empty directory/folder (zOS-Automation-Repository) created on my desktop.

git clone command

Microsoft Windows [Version 10.0.18363.1082]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\user1>git clone https://github.com/SubhasishSarkarIndia/JupyterNotebook.git C:\Users\user1\Desktop\zOS-Automation-Repository
Cloning into 'C:\Users\user1\Desktop\zOS-Automation-Repository'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 15 (delta 7), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (15/15), done.

Next, you want to automatically copy your four JCL job PDS members residing on z/OS as .txt files and place those files on your local machine. In this case, you'll use FTP. Place the following lines into a .txt file (let’s call the file SampleFTP.txt) and place the file on your desktop (or, for that matter, anywhere on your local Windows machine).

open nnn.nnn.nnn.nnn         
XXXXXX                       
ZZZZZZZZ                     
get 'HLQ1.HLQ2.JOBS(REXEXC00)' REXEXC00.txt
get 'HLQ1.HLQ2.JOBS(REXEXC01)' REXEXC01.txt
get 'HLQ1.HLQ2.JOBS(REXEXC02)' REXEXC02.txt
get 'HLQ1.HLQ2.JOBS(REXEXC03)' REXEXC03.txt
quit

Note: In the above code listing, nnn.nnn.nnn.nnn, XXXXXX, and ZZZZZZZZ should be replaced with the IP address of the z/OS system you want the files to be FTP’ed to, your TSO logon user ID, and password, respectively.

You will now create a Windows Batch Script file -- let's call the file SampleFTP.bat. You can place the file in the same location as SampleFTP.txt. The batch script automatically copies the z/OS JCL job .txt files and does a commit to the repo. The batch script should look like the following:

cd C:\Users\user1\Desktop
ftp -s:SampleFTP.txt
xcopy /y REXEXC00.txt "C:\Users\user1\Desktop\zOS-Automation-Repository"
xcopy /y REXEXC01.txt "C:\Users\user1\Desktop\zOS-Automation-Repository"
xcopy /y REXEXC02.txt "C:\Users\user1\Desktop\zOS-Automation-Repository"
xcopy /y REXEXC03.txt "C:\Users\user1\Desktop\zOS-Automation-Repository"
del REXEXC00.txt
del REXEXC01.txt
del REXEXC02.txt
del REXEXC03.txt
cd C:\Users\user1\Desktop\zOS-Automation-Repository
echo ((((( Some sample text being entered here solely for testing purposes ))))) %DATE% %TIME%> README.md
git add .
git commit -m "Required files updated."
git push

Now you can create your first "Freestyle project" item in Jenkins. This should serve as your initial Trigger Job. Click on the New Item link:

New Item link

New Item link

Select Freestyle project and enter a name for the project. Let's name the project "Trigger-Script." Then click OK.

Freestyle project

Freestyle project

Source Code Management

Source Code Management

Build Triggers

Build Triggers

The job build has been scheduled to be triggered every day at 03:15 p.m. local time (in this case, IST - Indian Standard Time).

Build Environment

Build Environment

Add build step

Add build step

Click on Add build step and select Execute Windows batch command from the dropdown list. And then for the command, you should enter the full path to the batch script file so that the batch script gets automatically triggered for execution.

Then, you can add a post-build action of "Editable Email Notification."

Enter a value in the Project From field -- this is the "Email from" field value in the actual email that you will eventually receive. You can select the value Attach Build Log from the Attach Build Log field dropdown list. Tick the checkbox corresponding to the Save to Workspace field.

Click on the Save button to save the job details.

4

Create your second "freestyle project" item in Jenkins

Now it's time to create your second "freestyle project" item in Jenkins. Let's name the project "Script-01."

Note: Before you proceed any further, make sure that you have the IBM z/OS Connector Jenkins Plugin installed.

Select Git as the Source Code Management value. For the Repository URL value, enter the same URL that you previously specified in the git clone command in your first freestyle project above. And enter your GitHub credentials in the Credentials field.

Source Code Management

Source Code Management - project #2

Add "Submit z/OS job" as a build step -- in fact, you should add two "Submit z/OS job" build steps. Then specify the LPAR SMF ID as the value of the Server field; this should be the LPAR on which you want the jobs to get executed. The Port field value can be 21. Use your TSO Login credentials in the Credentials field. For the first "Submit z/OS job" build step, specify "REXEXC00.txt" as the value for the Job file field and "0001" as the value for the MaxCC field.

Submit z/OS job build step #1

Submit z/OS job #1

For the second "Submit z/OS job" build step, specify "REXEXC01.txt" as the value of the Job file field and "0000" as the value for the MaxCC field.

Submit z/OS job build step #2

Submit z/OS job #2

The main objective here is to trigger the execution of the successor REXEXC01 job on z/OS if and only if the preceding REXEXC00 job has completed execution with either RC=0 or RC=1. And the REXEXC00 job has been coded to set a MaxCC equal to 1, and therefore REXEXC01 should always get triggered.

Finally, you can add a post-build action of "Editable Email Notification" then click the Save button to save the job details.

5

Create your third "freestyle project" item in Jenkins

Let's move on to the third "freestyle project" item in Jenkins. You can name the project "Script-02" and once again, you should select Git as the Source Code Management value.

Once again, add two "Submit z/OS job" build steps. For the first one, specify "REXEXC02.txt" as the value of the Job file field and "0001" as the value for the MaxCC field. For the second, specify "REXEXC03.txt" as the value of the Job file field and "0000" as the value for the MaxCC field.

The main objective here is to trigger the execution of the successor REXEXC03 job on z/OS if and only if the preceding REXEXC02 job has completed execution with either RC=0 or RC=1. However, the REXEXC02 job has been coded to set a MaxCC equal to 2, and therefore REXEXC03 should never get triggered. This should cause the Jenkins project/job "Script-02" to fail.

Finally, you can add a post-build action of "Editable Email Notification" then click the Save button to save the job details.

One of the main objectives of this POC project is to confirm that you can use Jenkins to automatically trigger the execution of jobs on z/OS such that the successor job gets triggered depending on the return code of the predecessor job.

6

Create the Jenkinsfile

You're going to need a Jenkinsfile, so let’s name it "Jenkinsfile.groovy." It should have the following content:

pipeline {
     agent any
     stages {
         stage('Script-01') {
             steps {
                 script {
                         build job: 'Script-01'
                     }
                 }
            }   
         stage('Script-02') {
             steps {
                 script {
                         build job: 'Script-02'
                     }
                 }
            }   
        }
    }

Next, upload the Jenkinsfile to your GitHub repository:

Jenkinsfile in GitHub repo

Jenkinsfile in GitHub repo

7

Build a multibranch pipeline in Jenkins

And now the final piece of the puzzle -- building a multibranch pipeline in Jenkins. Click on the New Item link and select Multibranch Pipeline. Enter a name for the pipeline -- let's call it "Multibranch-Pipeline-01" -- then click on OK.

Multibranch Pipeline name

Multibranch Pipeline name -- image 1

Add a GitHub source under Branch Sources. Add your GitHub credentials in the Credentials field. Select the option Repository HTTPS URL and key in as the Repository HTTPS URL field the same URL that you previously specified in the git clone command in your first freestyle project above.

Branch Sources

Branch Sources

Build Configuration, Scan Repository Triggers, and Orphaned Item Strategy

Build Configuration

Choose the Scan Repository Triggers interval option of 2 minutes; this means that the Jenkins pipeline will poll your GitHub repository every 2 minutes to check for any commits made to the repo. Whenever the initial Jenkins trigger job (named "Trigger-Script") does a commit to the GitHub repository, the multibranch pipeline triggers a build process as per the Groovy script present in the Jenkinsfile, and jobs start executing on z/OS. Thus, the multibranch pipeline triggers a scan of the GitHub repository every 2 minutes (the polling interval).

Health metrics

Health metrics

Click Save to save the Multibranch Pipeline details. And with that, you are now done with the development phase -- quite an achievement!

8

Test your automation project

Time to test what you have developed. Although the initial Jenkins trigger job (named "Trigger-Script") build process has been scheduled to be triggered automatically every day at 03:15 p.m. local time (in this case, IST - Indian Standard Time), you will do a manual trigger of the "Trigger-Script" build in order to test the entire POC.

Manually initiate build process

Manually initiate build process

Click on the highlighted icon shown in the screenshot above to manually initiate the “Trigger-Script” build process. The “Trigger-Script” job should do a commit to your GitHub repository.

Trigger-Script commits to repo

Trigger-Script commits to repo

The commit made to the repo should automatically trigger the pipeline build process. After the pipeline has completed executing, you can go ahead and check the console output.

Jenkins - click on the pipeline link

Jenkins - click on pipeline link

Jenkins - click on Build History

Jenkins - click on Build History

Jenkins - click on the Console Output icon

Jenkins - click on Console Output

Now, scroll down the content of the console output:

Script-01 console output 1

Script-01 console output - image 1

Script-01 console output 2

Script-01 console output - image 2

Script-01 console output 3

Script-01 console output - image 3

Script-01 console output 4

Script-01 console output - image 4

Script-01 console output 5

Script-01 console output - image 5

The "Script-01" job should finish successfully. Now let's view the console output for the "Script-02" job:

Script-02 console output 1

Script-02 console output - image 1

Script-02 console output 2

Script-02 console output - image 2

As explained earlier, the execution of the successor REXEXC03 job is triggered on z/OS if and only if the preceding REXEXC02 job has completed execution with either RC=0 or RC=1. However, the REXEXC02 job has been coded to set a MaxCC value equal to 2, and therefore REXEXC03 should never get triggered. This should cause the Jenkins project/job "Script-02" to fail. Therefore, everything has worked as expected until now!

Script-02 completed with status "FAILURE" - image 1

Script-02 completed with status "FAILURE"

Script-02 completed with status "FAILURE" - image 2

Script-02 completed with status "FAILURE"

Summary

As organizations become more and more agile, "mean time to market" becomes increasingly important as they work to achieve greater speed and quality. IBM Z is a critical component of the IT infrastructure in many enterprises, and should be a part of those organizations' CI/CD DevOps initiatives. This POC has demonstrated exactly that -- how to achieve a fully automated CI/CD pipeline for z/OS testing activities. If an organization wants to achieve a fully automated CI/CD DevOps pipeline, automating the testing infrastructure is a key part and this POC has clearly shown you how to achieve a fully automated z/OS testing CI/CD pipeline -- thereby enabling faster velocity with uncompromising quality.

If you have successfully completed this POC, you may want to take the next step and experiment further with building more automation capabilities with Zowe and Z Open Automation Utilities (ZOA Utilities). This will enable you to use the right technology, tools, and resources according to your specific use case(s). For more information, check out the following resources:

Create a fully automated CI/CD pipeline for z/OS testing

Create a fully automated CI/CD pipeline for z/OS testing