Build a pipeline with Jenkins, Dependency Based Build, and UrbanCode Deploy

This tutorial shows how to configure UrbanCode® Deploy (UCD) and Jenkins to build a pipeline that streamlines the development of applications up to the deployment phase. The sample application for this tutorial is GenApp, the IBM® standard CICS® application.

After the configuration steps, this tutorial shows how to quickly test the configuration by starting a Jenkins build that runs a Dependency Based Build (DBB) build. This build will create a version in the client UCD, upload the binary files to Artifactory, and deploy the application via UCD, using the binary files in Artifactory.

Note: This tutorial uses only one pipeline for all of the steps (build/package/deploy). However, in a real environment, this should be split into several pipelines depending on the step (development, staging, production, etc.).

Overall prerequisites

  • You need access to an Artifactory repository.
  • You must have a Jenkins server installed and configured.
  • The DBB toolkit and server must be installed. For more information, see Installing and configuring DBB in the IBM Dependency Based Build Knowledge Center.
  • UCD version 7.0.4 or later must be installed and configured. This tutorial is based on UCD version 7.0.5. For more information, see the IBM UrbanCode Deploy V7.0.5 documentation.
  • The GenApp application must be installed in your CICS environment and CMCI must be configured. You can download GenApp from the IBM Support website.

Estimated time

It should take you about 60-90 minutes to complete this tutorial.

Steps

Here are the steps required to complete this tutorial:

Part 1. Configure UrbanCode Deploy

  1. UCD prerequisites
  2. Install two required plugins
  3. Create a UCD component for the GenApp application
  4. Create a deployment process in the GenAppComponent component
  5. Describe the deployment process of the component
  6. Create a resource to deploy on the target
  7. Create a deployment configuration

Part 2. Configure Jenkins

  1. Jenkins prerequisites
  2. Set up your Git source repository
  3. Set up Jenkins
  4. Run the pipeline in Jenkins

Part 1. Configure UrbanCode Deploy

1. UCD prerequisites

The UrbanCode Deploy agent must be installed on z/OS®. The name of the agent used in this tutorial is “e2e-pipeline”.

Your UCD instance must be ready to use.

Then you need to open it. Follow the tutorial steps to configure your UCD instance and deploy the application in the pipeline context. The application, named GenApp, is a CICS application.

2. Install two required plugins

Install the following two plugins, which are required to deploy a CICS application and to push the binary files to Artifactory:

Click on the first link. Then, click Download and select I Agree to start downloading. Do the same with the second link.

In UCD, click the Settings tab. In the Automation panel, click Automation Plugins. Then click Load Plugin.

Select the downloaded plugins (.zip), one at a time, and click Submit.

You can see that the two plugins (CICS TS and the z/OS Utility to download external artifacts) are now displayed in the Automation plugins list.

3. Create a UCD component for the GenApp application

Now you need to declare the GenApp application as a component in UCD.

In UCD, click the Components tab and click Create Component.

In the wizard that opens, enter “GenAppComponent” in the Name field and select “z/OS” for the field Component Type.

Then click Save.

4. Create a deployment process in the GenAppComponent component

Once the UCD component is created, you must create a process in this component. A process describes a set of tasks that must be run to deploy the application. A typical set of tasks for a CICS application is to deploy the load modules to the target environment, to bind, and to make a new copy to refresh the updated load modules.

The following steps show you how to insert the tasks into the graphical interface of the process. At the end, the process will look like this. You can refer to it each time you insert a task.

Graphical interface tasks

You must start by creating the process. So, in the GenAppComponent component, click the Processes tab and click Create Process.

Creating the process

In the wizard that opens, enter “Deploy” in the Name field. Leave the default options. Make sure that the Process Type is Deployment.

Then click Save. The graphical interface of the Deploy process opens.

5. Describe the deployment process of the component

In the graphical interface of the Deploy process, you must now describe all the tasks that will be run to deploy the GenApp application. This deployment process is not specific to a target — it can be run in any z/OS environment.

When you insert a task, a rectangle is added to the graph.

Here’s some basic information about the graphical interface:

  • You can click Save in the toolbar when you modify the graph. You can then click Revert to go back to the last saved version of the graph.
  • To rearrange the display of the rectangles in the graph, you can slide the Autolayout cursor to the right in the toolbar.
  • You can manually set a link between two tasks. To do so, hover over the source task, click the arrow in the blue circle and, with the mouse button still pressed, go to the target task and release the button. This inserts a link between the two tasks.
5.1. Download the artifacts from Artifactory

The deploy phase comes after the build phase, which created a version in UCD and pushed the binary files to Artifactory. So the first task of this process is to retrieve these binary files.

In the Type to filter field on the left side of the graphical interface, copy and paste the words “Download Artifacts for zOS External Repo”.

Download artifacts

Then, drag the zOS ExternalArtifactsDownload line found under Artifact and drop it between the Start and Finish tasks.

Edit the properties of the task by clicking the edit icon on the top right corner of the rectangle and proceed as follows:

  • Keep “${p:extRepoURL}” in Repository URL. It represents the Artifactory repository that contains the binary files of the application. The value of this property will be assigned during the build and associated with the UCD version.
  • Enter “${p:artifactory.user}” in Repository User Name. It represents the user ID of the Artifact repository.
  • Enter “${p:artifactory.password}” in Repository Password. It represents the password of the user in the Artifact repository. Even if the value is hidden, it will be recognized.

As a UCD component can be deployed on any z/OS environment, the actual values of these last two properties will be assigned when a specific target is specified for the deployment.

Click OK.

5.2. Deploy Data Sets

This task deploys the binary files from Artifactory to the target. It deploys the load modules and the DBRMs (data set members created by Db2® for the z/OS precompiler) to PDS on z/OS. The DBRMs will be used in the bind process.

In the Type to filter field on the left side of the graphical interface, copy and paste the words “Deploy Data Sets”.

Then drag the Deploy Data Sets line found under zOS Utility and drop it between the Download Artifacts for zOS External Repo and Finish tasks.

You can keep all the default values of the properties. The Data Set Mapping property, with its default deploy.env.pds.mapping value, conveys that the load modules and the DBRMs will be deployed to PDS in the target environment.

The next tasks in the process will constitute two branches that originate from the Deploy Data Sets task:

  • The branch for the CICS load module process
  • The branch for the bind process

The processing for these two branches is run in parallel.

5.3. CICS load module tasks

The CICS application must recognize the deployed load modules. So, add the following two tasks:

  1. List the load modules to refresh

    This CICS task generates a string that lists the load modules to be refreshed. This list will be used in the next task.

    In the Type to filter field on the left side of the graphical interface, copy and paste the words “Generate Artifact Information”.

    Then, drag the Generate Artifact Information line found under zOS Utility and drop it between the Deploy Data Sets and Finish tasks.

    Edit the properties and proceed as follows:

    • Enter “Generate Load Module List” in the Name field. This name will be reused later.
    • Enter “/.*LOAD*/” in the Deploy Type Filter field. Only the binary files with a LOAD type (load module) in the package deployed in Artifactory will be considered.
    • Enter “${member},” (with the comma at the end) in the Template field. Clicking Template opens an editor where you can copy and paste the value. This value means that a line with the name of the member will be generated for each member present in Artifactory. Click Save to go back to the properties.

    Then click OK.

  2. New copy

    This CICS task reloads the load modules generated in the preceding step.

    In the Type to filter field on the left side of the graphical interface, copy and paste the words “New copy resources”.

    Next, drag the New copy resources line found under CICS TS and drop it between the Generate Load Module List and Finish tasks.

    Edit the properties and enter “${Generate Load Module List/text}” in Resource Name List. Clicking this field opens an editor where you can copy and paste the value. This value means that the list produced in the preceding step will be output as text. Click Save to go back to the properties.

    Make sure that the Resource Type is “Program.”

    Clicking Precondition opens an editor where you can copy and paste the following lines:

    properties.get("Generate Load Module List/text") !== undefined &&
    properties.get("Generate Load Module List/text") !== null &&
    properties.get("Generate Load Module List/text") != ""
    

    Click Save to return to the properties.

    Leave the default values of the other properties.

    If you click Show Hidden Properties, you can see that this task will require information to log into CICS. The values will be assigned in the deployment environment.

    Click OK.

5.4. Bind tasks

When Db2 load modules are deployed with new DBRMs, they must be bound again for the Db2 database in use. So, add the following two tasks:

  1. Generate artifact information

    If a COBOL that contains Db2 statements is modified, the load module must be bound again with the configured Db2 package.

    In the Type to filter field on the left side of the graphical interface, copy and paste the words “Generate Artifact Information”.

    Next, drag the Generate Artifact Information line found under zOS Utility and drop it at the same level as Generate Load Module List. Then, connect it to the Deploy Data Sets task. To do this, hover over the Deploy Data Sets task, click the arrow in the blue circle and, with the mouse button still pressed, go to the Generate Artifact Information task and release the button. This will insert a link between the two tasks.

    You should then see the second branch starting from the Deploy Data Sets task.

    Edit the properties and proceed as follows:

    • Enter “Generate Bind Package” in the Name field.
    • Enter “/.*DBRM*/” in the Deploy Type Filter field. Only the binary files with a DBRM type in the package deployed in Artifactory will be considered.
    • Enter a SQL text fragment that the JCL will use to bind each member. It will bind the package (information from the Db2 database configured on z/OS), the DBMR member, the qualifier (Db2 information), and the owner (user who will run the bind). The other lines of the fragment are required for the Db2 bind. Click Template to open an editor where you can copy and paste the following lines to overwrite the default line:
    BIND  PACKAGE(${p:db2.target.collId}) +
       MEMBER(${member}) +
       LIBRARY('${dataset}') +
       QUALIFIER(${p:db2.target.qualifier}) +
       OWNER(${p:jes.user}) +
       ACTION(REPLACE) +
       ISOLATION(CS) +
       RELEASE(COMMIT) +
       ENCODING(EBCDIC)
    

    Click Save to go back to the properties.

    Then click OK.

  2. Submit the job on z/OS

    In the Type to filter field on the left side of the graphical interface, copy and paste the words “Submit Job”.

    Then drag the Submit Job line, found under zOS Utility, and drop it after the Generate Bind Package task. Connect it to Generate Bind Package task and, since this is the last task, connect it to New copy resources.

    Enter the JCL lines in the JCL field. This will bind the new DBRMs. Values will be assigned to specific properties later:

    • ${p:db2.hlq} — where Db2 HLQ is installed
    • ${p:db2.target.sqlid} — Db2 on z/OS
    • ${Generate Bind Package/text} — the text generated in the preceding step will be produced

    Clicking JCL opens an editor where you can copy and paste the following lines:

    //BINDPKG  JOB 'DBB-PKGBIND',MSGLEVEL=(1,1),MSGCLASS=R,NOTIFY=&SYSUID
    //*ROUTE PRINT @JCLPRINT@
    //JOBLIB   DD  DISP=SHR,
    //             DSN=${p:db2.hlq}.SDSNEXIT
    //         DD  DISP=SHR,
    //             DSN=${p:db2.hlq}.SDSNLOAD
    //*******************************************
    //* PKGBIND
    //* Step bind packages
    //*******************************************
    //**BEGIN
    //PKGBIND EXEC PGM=IKJEFT01,DYNAMNBR=20,COND=(4,LT)
    //SYSTSPRT DD  SYSOUT=*
    //SYSPRINT DD  SYSOUT=*
    //SYSUDUMP DD  SYSOUT=*
    //SYSIN    DD  DUMMY
    //SYSTSIN  DD  *
    DSN SYSTEM(${p:db2.target.sqlid})
    ${Generate Bind Package/text}
    END
    /*
    //*END
    

    Leave the default values for the other properties. If you click Show Hidden Properties, you can see the host where the job will run, the JES job monitor port, the user name, and password.

    Click Save to return to the properties.

    Then click OK.

    If you haven’t saved the graph yet, you should do so now.

6. Create a resource to deploy on the target

This task configures the target of the deployment. Note that a z/OS agent has been created and configured in the target environment. UCD communicates with it when a deployment is requested on this z/OS.

The resource consists of the resource itself, an agent, and a component.

Click the UCD Resources tab and then Create Top-level Group.

In the wizard that opens, enter the resource name “GenAppResource” and click Save.

Then add an agent to this resource: On the GenAppResource line, click on the ellipsis () and select Add Agent.

In the wizard that opens, expand the Agent list and select the agent where you want to deploy GenApp. In our example, the agent is e2e-pipeline. Click Save. The agent is now displayed in the Resource Tree, nested under GenAppResource.

Next, you need to associate a component with the agent. To do so, on the e2e-pipeline line, click on the ellipsis () and select Add Component.

In the wizard that opens, expand the Component list and select the GenAppComponent. Click Save. The component is now displayed in the Resource Tree, nested under the e2e-pipeline. This is what you should see:

Resource Tree

7. Create a deployment configuration

7.1. Create an application

Click the UCD Applications tab. Click Create Application, expand the list, and select New Application.

In the wizard that opens, enter the application name “GenApp-Deploy” and click Save.

You should see the message “No environments found.” Don’t worry, you’ll create an environment shortly.

7.2. Associate a component with the application

In the GenApp-Deploy application, click the Components tab. Then click Add Component.

In the wizard that opens, expand the Select a Component list and select GenAppComponent. Then click Save.

7.3. Define a process to install this component

In the GenApp-Deploy application, click the Processes tab. Then click Create Process.

In the wizard that opens, enter the name of the process, “Deploy”, then click Save.

The graphical interface of the process should open.

Drag Install Component (under Application Steps) and drop it between the Start and Finish tasks.

Install component

Edit the properties and make sure that GenAppComponent is selected in the Component property. This means that the process will deploy this component.

Click OK in the properties.

Then click Save on the process.

7.4. Create an environment for the application

In this task, you will make the final link with the deployment target — so you need to create a development environment.

In the GenApp-Deploy application, click the Environment tab. Then click Create Environment.

In the wizard that opens, enter a name for the environment: “Development”, then click Save.

7.5. Assign resources to the environment

In the list of the environments for the GenApp-Deploy application, click Development. Then click Add Base Resources.

In the Add Resource to Environment dialog box, select the top-level GenAppResource item and then click Save.

Return to the list of the environments by going back up one level. To do this, click Environments in the breadcrumb trail under the UCD tabs.

7.6. Assign values to the environment properties

In the list of the environments for the GenApp-Deploy application, click Development.

In the Development environment notebook, click the Configuration tab. Then click Environment Properties on the left part of the tab.

Click Batch Edit and copy-paste the following lines to the text area, then click Save:

cics.cicspex=YOUR_CICS_CICSPEX
cics.cmciport=YOUR_CICS_CMCIPORT
cics.host=YOUR_CICS_HOST
cics.username=YOUR_CICS_USER
deploy.env.pds.mapping=*.LOAD, “YOUR_PDS_FOR_LOAD_MODULES“ \n\
*.DBRM, “YOUR_PDS_FOR_DBRM“ 
db2.hlq=YOUR_DB2_HLQ
db2.target.sqlid=YOUR_DB2_SQLID
jes.user=YOUR_ZOS_USERID
jes.host=YOUR_ZOS_HOST. As the access is local, you can put “\:\:1” for IPV6 or “localhost” for IPV41.
jes.monitor.port=YOUR_ZOS_JOBMONITOR_PORT (6715 if not customized)
db2.target.collId=YOUR_GENAPP_DB2_COLLECTION_ID
db2.target.qualifier=YOUR_GENAPP_DB2_QUALIFIER
artifactory.user=YOUR_ARTI_USER

Close the Alert box.

Click the Back to table mode button above the text area.

Next, you need to add three properties whose values will be hidden because they correspond to passwords.

  1. Click Add Property and enter “cics.password” in the Name field. Select the Secure checkbox to hide the value and enter “YOUR_CICS_PASSWORD” in the Value field. Then click Save.
  2. Click Add Property again. Enter “artifactory.password” in the Name field. Select the Secure checkbox to hide the value and enter “YOUR_ARTI_PASSWORD” in the Value field. Then click Save.
  3. Click Add Property again. Enter “jes.password” in the Name field. Select the Secure checkbox to hide the value and enter “YOUR_ZOS_PASSWORD” in the Value field. Then click Save.

Congratulations, you’ve configured UCD!

Part 2. Configure Jenkins

For this tutorial, Jenkins already contains the plugins required to run the pipeline. Now you need to configure Jenkins to create the pipeline that will deploy the application by using the UCD instance that you’ve configured. The tutorial is based on Jenkins version 2.222.1.

1. Jenkins prerequisites

You must have the following Jenkins plugins in your Jenkins instance:

  • Durable Task (version >= 1.29)
  • Git
  • Git Client
  • Pipeline
  • Pipeline Utility Steps
  • SSH Build Agents
  • GitHub (optional — only if you want to test webhook)

2. Set up your Git source repository

At the time this tutorial is being written, the source code of GenApp is only available as MVS content through IBM Support. You need to extract the code source on MVS™ (see previous link) and add it back to your Git.

  1. Fork this repository: https://github.com/IBM/dbb-pipeline.
  2. Add the GenApp source code in the “cics-genapp” folder of your forked repository with the following structure:

    - cics-genapp
      - base
        - src
          - BMS    
            - Duplicate the contents of the PDS 'HQL'.BMS
              extracted on MVS from SOURCE.XMIT
          - COBOL
            - Duplicate the contents of the PDS 'HQL'.COBOL
              extracted on MVS from SOURCE.XMIT   
          - COPY
            - Duplicate the contents of the PDS 'HQL'.COPY
              extracted on MVS from SOURCE.XMIT
    
  3. Edit the “cics-genapp/application-conf/datasets.properties” file according to your targeted z/OS.

  4. Commit and push your changes into your forked repository.

3. Set up Jenkins

Log into your Jenkins server. The Jenkins dashboard should open:

Jenkins dashboard

3.1. Install the required UCD plugin for the Jenkins pipeline

The UCD Jenkins Pipeline plugin must be installed manually. So, go to the Jenkins Pipeline page on UrbanCode and click Download. Select I Agree to start downloading.

Once the plugin is downloaded, go back to Jenkins. In the Jenkins dashboard, click Manage Jenkins and then select Manage Plugins.

In the browser that opens, select the Advanced tab.

In Upload Plugin, browse to the UCD plugin that you’ve just downloaded and click Upload. The plugin is then uploaded and installed.

3.2. Create the credentials for the pipeline

Go back to the Jenkins dashboard and click Credentials. In the table, click Jenkins.

In the table on the System page that opens, click Global credentials (unrestricted).

In the left part of the Global credentials (unrestricted) page that opens, select Add Credentials for each credential that you create:

  1. Add the credential to connect the Jenkins agent to z/OS. Make sure that Username with password is selected in the Kind dropdown list. Enter the z/OS SSH user name that will own the Jenkins agent process in Username. Enter the z/OS password of the SSH user in Password. Enter zosuser_id in ID and z/OS User in Description. Then click OK.
  2. Add the credential to upload the binary files to an Artifactory repository. Make sure that Username with password is selected in the Kind dropdown list. Enter artifactory_id in ID, “Artifactory” in Description, and your own credentials in Username and Password. Then click OK.

You should now see the following two credentials:

Jenkins credentials

3.3. Configure Jenkins for UCD

Go back to the Jenkins dashboard and click Manage Jenkins. Then click Configure System.

In the configuration page that opens, scroll down to IBM UrbanCode Deploy Pipeline Plugin Configuration and click Add to add a UCD server. Enter UrbanCodeE2EPipeline in Profile Name, the URL of your UCD server in IBM UrbanCode Deploy URL, and the user and password of your UCD instance in User and Password. Then click Save.

Note: In a test configuration, you may need to select Trust All Certificates. However, for security purposes, all the certificates must be authentic in a production environment.

3.4. Create a Jenkins agent on z/OS

Go back to the Jenkins dashboard and click Build Executor Status.

In the Nodes page that opens, you can see a default agent named master. However, you will need to create a z/OS agent. To do so, click New Node. Enter “e2e-pipeline” in the Name field and select Permanent Agent. Then click OK.

In the configuration page that opens, enter the following information:

  • Enter the remote directory in Remote root directory. It can be “/var/jenkins/agent/e2e-pipeline,” but it varies according to the environment. It is the working directory of the agent.
  • Enter “e2e-pipeline” in Label.
  • Select Launch agents via SSH in Launch method.
  • Enter the hostname of your z/OS in Host.
  • Select z/OS User, the credential you created in a previous step, in Credentials.
  • Select Non verifying Verification Strategy in Host Key Verification Strategy because this is a test environment. However, bear in mind that in a production environment you will have to select Known hosts file Verification Strategy for security reasons.
  • Click Advanced to specify the following advanced parameters:
    • Keep the default value, 22, in Port. However, if your z/OS SSH port is different, you must change this value.
    • Enter “/usr/lpp/java/J8.0_64/bin/java” in JavaPath. This is where the Java™ binary files are installed.
    • Enter “-Xquickstart -Dfile.encoding=UTF-8 -Xnoargsconversion” in JVM Options.
    • Enter the following in Prefix Start Agent Command:
      . /usr/lpp/IBM/dbb/conf/gitenv.sh && export JAVA_HOME=/usr/lpp/java/J8.0_64 && export IBM_JAVA_ENABLE_ASCII_FILETAG=ON && export GIT_CONFIG_NOSYSTEM=0 && env &&
      
      Note that the gitenv.sh script is shipped with Dependency Based Build.
    • Enter ” -text” (with a whitespace at the beginning) in Suffix Start Agent Command.
  • Select Environment properties and add the following properties:
    • Enter “DBB_HOME” in Name and “/usr/lpp/IBM/dbb” in Value. This specifies where Dependency Based Build is installed on z/OS.
    • Enter “DBB_URL” in Name and the URL of the DBB server in Value. This specifies the URL of the Dependency Based Build server.
    • Enter “DBB_HLQ” in Name and the z/OS DBB HLQ in Value. This specifies the HQL of the PDS that the DBB build will create on z/OS.
    • Enter “ARTIFACTORY_URL” in Name and the URL of the Artifactory server in Value. This specifies the URL of the Artifactory server.
    • Enter “ARTIFACTORY_REPO_PATH” in Name and the Artifactory repository path in Value. This specifies the Artifactory repository path where binary files will be uploaded.
    • Enter “UCD_BUZTOOL_PATH” in Name and the location of buztool.sh in Value. It specifies the absolute location of buztool.sh on z/OS.
  • Select Tool Locations and declare a wrapper shipped with DBB /usr/lpp/IBM/dbb/bin/git-jenkins.sh in Home.

    Note: If your Jenkins Git plugins are recent, you might need to use the git-jenkins2.sh wrapper scripts available on GitHub.

Click Save to save the agent configuration. The agent is then launched automatically.

3.5. Create a Jenkins project

This project will constitute the pipeline.

Go back to the Jenkins dashboard and click New Item to create a project. Enter “Development” as the name, select Pipeline, and click OK.

The configuration notebook of the pipeline opens. Enter the following information in the Pipeline tab:

  • Select Pipeline script from SCM in Definition.
  • Select Git in SCM because the pipeline will use a Jenkins file that will be installed in Git. Enter the URL of the Git repository you previously forked in Repository URL.

    Note: You will use the HTTPS protocol for Git. On z/OS, you use Rocket Git and, to be able to use the HTTPS protocol, you must have a .gitconfig file in the home location of the user who owns the Jenkins agent process with the following contents:

         [http]
          sslVerify = false
    
  • Put “cics-genapp/Jenkinsfile” in Script Path. This Jenkins script describes a pipeline. It makes initializations, clones the repositories, starts the DBB build, creates the version that will be pushed to UCD and Artifactory, and calls UCD to deploy the application.

Click Save.

4. Run the pipeline in Jenkins

Go back to the Jenkins dashboard. You should see that the e2e-pipeline node is ready to run. You can click the Development pipeline and click Build now.

The Stage View part of the Jenkins development pipeline displays the duration of each constituting stage of the pipeline:

Run the pipeline in Jenkins

Note: This pipeline is a demonstration flow. In a real environment, some stages should be part of another pipeline.

If you want your pipeline to be triggered by a webhook, you can configure the Git webhook for your organization and your Git provider.

Summary

This tutorial has helped you understand and set up Jenkins CI facilities to integrate z/OS platforms into a unique solution. With the integration with UCD, you are now able to deploy a CICS application that uses Artifactory as the binary repository. You can now extend your Jenkins CI pipeline with z/OS actions to include unit testing, code review, and code coverage.