Build a pipeline with GitLab CI, IBM Dependency Based Build, and IBM UrbanCode Deploy

GitLab CI (Continuous Integration) is a service included in GitLab. It brings the DevOps best practices into the application development phase. When developers deliver code modifications, it triggers a pipeline that automatically builds, tests, and deploys the application into the target environment.

This tutorial focuses on how you can use GitLab CI to build a Continuous Integration / Continuous Deployment (CI/CD) pipeline. In GitLab, you can define a project that combines a Git repository with CI/CD pipeline features.

In this tutorial, you will learn how to configure GitLab CI for a CI/CD pipeline that includes builds with IBM Dependency Based Build (DBB), packaging and deploy phases with IBM UrbanCode Deploy (UCD), automated unit testing with the zUnit feature of IBM Developer for z/OS (IDz), and code coverage and code review with IDz.

This tutorial assumes that you have already configured UCD and therefore does not detail this step.

The sample application for this tutorial is GenApp, the IBM standard CICS application.

After the configuration steps, this tutorial shows how to quickly test the configuration by starting a GitLab CI build that runs a DBB build. This build creates a version with the client UCD, uploads the binary files to JFrog Artifactory, and deploys the application through UCD, using the binary files in Artifactory.

Note: This tutorial uses only one pipeline for all the steps (build, test, code coverage, IDz code review, package, deploy). However, in a real environment this should be split into several pipelines depending on the step (development, staging, production, and so on).

Prerequisites

You need to make sure that the following prerequisites are fulfilled:

• An account on the public GitLab server or on a local GitLab server. This tutorial is based on version 13.9.0 of the public GitLab server (https://gitlab.com).

• IBM DBB toolkit (version 1.0.9.ifix1 or later). See Installing and configuring DBB. It is highly recommended to install the latest PTF for the removal of the blank character in the first column of IDz code review reports.

• UCD version 7.0.4 or later. It must be installed and configured. This tutorial is based on UCD version 7.0.5. For more information, see the IBM UrbanCode Deploy Version 7.0.5 documentation.

  Note: You must have configured UrbanCode Deploy, as explained in Part 1. Configure UrbanCode Deploy of the tutorial “Build a pipeline with Jenkins, Dependency Based Build, and UrbanCode Deploy."

• The GenApp application. It must be installed in your CICS environment and CICS management client interface (CMCI) must be configured. You can download GenApp from the IBM Support website.

• Access to an Artifactory repository server.

• (Optional) For zUnit, you need IBM z/OS Dynamic Test Runner. See IBM z/OS Dynamic Test Runner Host Configuration Guide. A detailed technical paper titled Integrating IBM zUnit Testing into an open and modern CI/CD pipeline is available in the IBM Support website to help you start with zUnit testing in the context of a pipeline.

• (Optional) For zUnit and code coverage, you need IBM z/OS Debugger (version 15.0.x or later). See Overview of IBM z/OS Debugger.

• (Optional) For IDz code review, you need IBM z/OS Source Code Analysis (version 15.0.0 or later). See Program Directory for IBM z/OS Source Code Analysis.

• (Optional) For zUnit, xsltproc is used on the distributed system. This binary file is required to perform Extensible Stylesheet Language Transformation (XSLT) of the zUnit test result to a format that the GitLab CI can understand. Most of the Linux distributed systems provide this binary file.

Estimated time

It should take you about 60 minutes to complete the steps in this tutorial. However, this estimated time does not include the time required to configure UCD (as mentioned in the Prerequisites section).

Steps

Here are the steps required to build a CI/CD pipeline using GitLab CI.

Step 1. Create a group in GitLab

Step 2. Create the runners in the GitLab group

1. Install the GitLab runner
2. Configure a runner for z/OS
3. (Optional) Configure a runner for Linux
4. Display the runners in the GitLab group

Step 3. Create projects in the GitLab group

1. Import the dbb GitHub repository
2. Import the zAppBuild project
3. Display the new projects in the GitLab group

Step 4. Configure environment variables in the GitLab group

Step 5. Import the dbb-pipeline GitHub repository

Step 6. Configure the GenApp application

1. Add the application source file
2. Modify the datasets.properties file

Step 7. Create the CI/CD pipeline

Step 8. Modify a COBOL file to start another pipeline

Step 1. Create a group in GitLab

A group in GitLab is a collection of projects. This group will contain the GitLab projects that will be created by importing the existing DBB GitHub repositories.

In the GitLab menu bar, click New > New Group.

In the Create Group wizard that opens, enter the name of the group and click Create group.

Step 2. Create the runners in the GitLab group

Runners are the agents that interact with GitLab CI on the operating systems.

1. Install the GitLab runner

You must install the GitLab runner version 13.8.0 or later according to your operating system. For this tutorial, it is installed on a Linux system, but it could also be installed on Microsoft Windows, Linux for IBM Z, or IBM z/OS Container Extensions (IBM zCX).

To learn more about installing GitLab runners, refer to the tutorial “Integrate IBM z/OS platform into continuous integration pipelines with GitLab”.

After you’ve installed the GitLab runner, you must register your own runners to run the CI/CD jobs that come from GitLab and optionally send the result back to the GitLab server. So, you must set up the communication between the GitLab server and the system to which the runner is connected.

2. Configure a runner for z/OS

In a z/OS environment, a limitation is that it is necessary to use a runner that uses a Secure Shell (SSH) executor. Be sure to apply the prerequisites as detailed in the tutorial “Integrate IBM z/OS platform into continuous integration pipelines with GitLab.”

In the system where you installed the GitLab runner, you must register a runner with the gitlab-runner register command line by specifying the following elements:

1. Specify the GitLab instance URL: https://gitlab.com. This is the URL of the GitLab server.

2. Specify the registration token that is associated with the group. To get this token, go back to the group you created in GitLab and click Settings > CI / CD.

   

   View image larger

   Click Expand on the Runners line. Copy the registration token provided in the Group runners section.

   

   View image larger

   Paste the value of the registration token when prompted.

3. Enter a description for the runner: SSH executor on the targeted z/OS.
4. Enter a tag for the runner: zos-ssh. So, when you create a pipeline you will be able to specify that a job will run on this runner.
5. Enter a type of executor: ssh. The executor determines the environment where each job runs. To target a z/OS environment, the only option is to use SSH.
6. Enter the SSH server address: your z/OS hostname. This is the IP address of the z/OS server that you will log into with the SSH protocol.
7. Enter the SSH server port. This is generally 22 but it can vary according to your environment.
8. Enter the SSH user who will connect through SSH.
   Reminder: This user must have Bash as default shell.
9. Enter the SSH password. If you do not want to enter the password, you can leave it blank and use an SSH private key instead.

Now that the runner for z/OS is successfully configured, you can configure another runner for Linux if you want to push your DBB build log and zUnit back to the GitLab server.

3. (Optional) Configure a runner for Linux

If you want to push the build log and zUnit test back to the GitLab server, you’ll need to configure this second runner because there is a limitation for an SSH executor on z/OS. You also need to set the GITLAB_REPORT_UPLOAD variable to true in the GitLab pipeline (as mentioned later in this tutorial). The build log files and the zUnit test results from the CI/CD pipeline cannot be published on the GitLab server with the z/OS SSH executor. So, the only way to publish the logs and the test results is to start a job that runs on a distributed system through a shell-type runner.

In the system where you installed the GitLab runner, you must register a runner with the gitlab-runner register command line by specifying the following elements:

1. Specify the GitLab instance URL: https://gitlab.com.
2. Enter the registration token that is associated with the group. This is the same token as for the first runner.
3. Enter a description for the runner: Shell executor on runner OS.
4. Enter a tag for the runner: dist-shell.
5. Enter a type of executor: shell.

The runner for Linux is now successfully configured.

4. Display the runners in the GitLab group

If you refresh the page of the CI/CD group settings, you can see that the two runners you just created are available and are ready to be used by your group.

View image larger

Step 3. Create projects in the GitLab group

You must create projects in the GitLab group to store the scripts that build the GenApp application. You need not create these projects from scratch. Instead, you can simply import the corresponding DBB GitHub public repositories. You need to import the following two GitHub repositories:

• dbb, which contains the scripts that are necessary to run the DBB builds.

• zAppBuild, which contains the Groovy scripts that are necessary to build a COBOL application in z/OS.

1. Import the dbb GitHub repository

1.1 Create the dbb GitLab project

In the GitLab menu bar, click New > New project.

View image larger

On the Create new project page, click Import project.

View image larger

On the Import project page, click Repo by URL.

View image larger

You should then see the page where you can enter the details of the repository to import.

View image larger

In the Git repository URL field, copy and paste the URL of the public IBM DBB GitHub repository: https://github.com/IBM/dbb.

In the Project name field, enter dbb.

In the Project URL field, open the drop-down list and select the GitLab group that you have created.

Then, click Create project.

GitLab imports the dbb repository from GitHub and creates the dbb project in the new GitLab group, as you can see in the following screen capture.

View image larger

1.2 Set up the pipeline

Now, you must set up the pipeline to be able to run the DBB builds for the GenApp application.

In the DBB project that you just imported, you must create a CI/CD pipeline. So, click Set up CI/CD.

View image larger

GitLab CI creates a file named .gitlab-ci.yml. This file controls the pipeline.

You must copy and paste the following code for this file.

stages:
  - Dependencies
Push DBB:
  stage: Dependencies
  script:
  - |
    echo "Push DBB project to USS"
  tags:
    - zos-ssh
  only :
    variables:
      - $CI_PIPELINE_SOURCE == 'pipeline'

In GitLab, the code looks as shown in the following figure.

View image larger

In this code, the Dependencies stage means that you will be able to clone the DBB repository when needed. The zos-ssh tag indicates the runner to be used. The only variable $CI_PIPELINE_SOURCE == 'pipeline' means that this pipeline can be started only from another pipeline. So, for example, it will not start if you modify a file in this project.

Click Commit changes.

2. Import the zAppBuild project

Now, you must import the other project,zAppBuild, which contains the Groovy scripts to build a z/OS application.

2.1. Create the zAppBuild GitLab project

The explanations given to create the dbb project also apply here, except for the following values on the Import project page:

• In the Git repository URL field, copy and paste the URL: https://github.com/IBM/dbb-zappbuild.
• In the Project name field, enter dbb-zappbuild.
• In the Project URL field, open the drop-down list and select the GitLab group that you have created.

Then, click Create project.

GitLab imports the dbb-zappbuild repository from GitHub and creates a dbb-zappbuild project in your GitLab group.

2.2. Set up the pipeline

Now, you must set up the pipeline so you can run the Groovy scripts for the GenApp application.

So, in the zAppBuild project that you just imported, you must create a CI/CD pipeline. Click Set up CI/CD.

In the .gitlab-ci.yml file, copy and paste the following code:

stages:
  - Dependencies
Push zAppBuild:
  stage: Dependencies
  script:
  - |
    echo "Push zAppBuild project to USS"
  tags:
    - zos-ssh
  only :
    variables:
      - $CI_PIPELINE_SOURCE == 'pipeline'

Click Commit changes.

3. Display the new projects in the GitLab group

In the left panel, expand Group overview and click Details. You should see that your GitLab group now contains the two imported projects.

View image larger

The next step is to add environment variables to the GitLab group to manage the passwords needed for your pipeline.

Step 4. Configure environment variables in the GitLab group

You must configure environment variables that will be used for the GenApp application. These variables correspond to passwords that will be required to log into Artifactory and UCD. This way, the passwords will be hidden.

So, navigate to the CI/CD Settings of the GitLab group and click Add Variable in the Variables section.

View image larger

The Add variable wizard opens.

View image larger

You need to create the variable that corresponds to the Artifactory password. Enter GROUP_ARTI_PASSWORD in the Key field and the password in the Value field. In the Flags section, clear the Protect variable checkbox and select the Mask variable checkbox. Then, click Add variable.

Next, you need to create the variable that corresponds to the UrbanCode Deploy password. Enter GROUP_UCD_PASSWORD in the Key field and the password in the Value field. In the Flags section, clear the Protect variable checkbox and select the Mask variable checkbox. Then, click Add variable.

You can see that the two variables are now configured in the GitLab group.

View image larger

Next, you must import the GitHub repository that contains examples of CI/CD pipeline scripts integrated with IBM DBB.

Step 5. Import the dbb-pipeline GitHub repository

This GitHub repository contains examples of CI/CD pipeline integrated with DBB. You must import it and create a project in your GitLab group.

The explanations given to create the dbb project also apply here (see Create_the_dbb_GitLab_project), except for the following values on the Import project page:

• In the Git repository URL field, copy and paste the URL of the public IBM DBB GitHub repository that contains the pipeline scripts: https://github.com/ibm/dbb-pipeline.
• In the Project name field, enter dbb-pipeline.
• In the Project URL field, open the drop-down list and select the GitLab group that you have created.

Then, click Create project.

GitLab imports the dbb-pipeline repository from GitHub and creates the dbb-pipeline project in your GitLab group, as you can see in the following screen capture.

View image larger

Step 6. Configure the GenApp application

Before you set up the CI/CD pipeline for the dbb-pipeline project, which will build the GenApp application, you must add the GenApp source files and modify a properties file.

1. Add the application source file

At the time this tutorial is being written, the source code of GenApp is available only as IBM MVS content through the IBM Support. From IBM Support, you need to extract the source code on MVS and add it back to the dbb-pipeline project.

Add the GenApp source files to the cics-genapp folder of your dbb-pipeline repository with the following structure:

- cics-genapp
  - base
    - src
      - BMS    
        - Duplicate the contents of the PDS 'HQL'.BMS
          extracted on MVS from SOURCE.XMIT
      - COBOL
        - Duplicate the contents of the PDS 'HQL'.COBOL
          extracted on MVS from SOURCE.XMIT   
      - COPY
        - Duplicate the contents of the PDS 'HQL'.COPY
          extracted on MVS from SOURCE.XMIT

Note: The transferred files must have their corresponding extensions in the Git repository. The extension is .cbl for the COBOL files, .bms for the BMS files, and .cpy for the COPY files.

2. Modify the datasets.properties file

You must adapt this file to your z/OS environment.

In your dbb-pipeline repository, under the cics-genapp folder, open the application-conf subfolder. Then, open the datasets.properties file.

Change the default properties to adapt them to your environment. Now you are ready to set up and run the CI/CD pipeline that integrates GitLab CI.

Step 7. Create the CI/CD pipeline

With the IDE of your choice, clone the dbb-pipeline project you created before in GitLab, locate and edit the .gilab-ci.yml.sample file, which contains the CI/CD pipeline. Rename this file as .gilab-ci.yml and adapt the pipeline global variables to your environments.

• By default, the zUnit, Code Coverage, Code Review, and GitLab reporting features are disabled. You can enable them if you have configured the related optional tasks mentioned in this tutorial by setting the following variables to true:

  
    RUN_ZUNIT_TEST: "false"
    RUN_CODE_COVERAGE: "false"
    RUN_IDZ_CODE_REVIEW: "false"
    GITLAB_REPORT_UPLOAD: "false"
  

  Show more
• Adapt other variables mentioned in the .gilab-ci.yml file according to your environments.
• In the dbb and zAppBuild jobs, replace the YOUR_GIT_LAB_GROUP pattern with your GitLab group.
• Then, commit and push the changes to your dbb-pipeline GitLab project.

This commit will trigger a full build as it is the first build.

You can follow the pipeline execution in the dbb-pipeline project. Open the GitLab CI server for your project and click CI/CD > Pipelines.

View image larger

You can see that the first execution of the pipeline is pending.

View image larger

Click pending to see the sequence of steps as specified in the .gilab-ci.yml file. When the pipeline completes the steps, its status turns to passed.

View image larger

If you click a pipeline, you can see the sequence of steps.

View image larger

You can click each step to see its details:

1. In the Dependencies step, the two dependent repositories (dbb and zAppBuild) needed for the build are cloned.
2. In the Build step, the DBB build is run.
3. (Optional) In the Codereview step, the IDZ Code Review is run.
4. (Optional) In the Report step, the build log files and the zUnit report are uploaded to GitLab CI. You can click Browse to see the reports.

Note: The UCD steps that are included in the pipeline are described in the tutorial “Build a pipeline with Jenkins, Dependency Based Build, and UrbanCode Deploy."

Step 8. Modify a COBOL file to start another pipeline

With the IDE of your choice, modify cics-genapp/base/src/COBOL/LGICDB01.cbl and commit your change to GitLab CI.

A new pipeline is run. You can click it to see the details.

View image larger

If you click DBB build, you can see that a DBB incremental build has run.

In the following screen capture, you can see that only the modified file (LGICDB01.cbl) was built and that the zUnit tests were successful.

View image larger

Optionally, if you select the CodeReview step, you can see that the Code Review was successful on z/OS.

View image larger

Optionally, you can do the same in the Report step.

View image larger

Click Browse to view the results of the job execution.

View image larger

You can click buildReport.html to see the result of the DBB build, or the .log files to see the result of the COBOL programs compilation. If you click the .pdf file, you can open the PDF that contains the Code Review report. A sample is shown in the following screen capture:

View image larger

You can select the Package step and see that it was successful.

View image larger

You can select the Deploy step and see that it was successful.

View image larger

Summary

This tutorial has helped you understand and set up GitLab CI facilities to integrate z/OS platforms into a unique solution. You can now build a CI/CD pipeline that’s orchestrated with GitLab CI for a CICS application with Dependency Based Build (DBB) and UrbanCode Deploy (UCD). You can also extend your GitLab CI pipeline with z/OS actions to include unit testing, code review, and code coverage.