Overview

Skill Level: Intermediate

Requires basic understanding of IBM Cloud Private and Kubernetes

This recipe describes the steps necessary to install the Jenkins Long Term Support docker image in your IBM Cloud Private cluster. After running this recipe you are able to access Jenkins through the Ingress controller.

Ingredients

You need to have

  • IBM Cloud Private installed
  • root access to the IBM Cloud Private nodes
  • administrative rights within the IBM Cloud Private instance

Note: the steps described in this tutorial are tested against an High Available deployment with 3 master and 3 worker nodes.

Step-by-step

  1. Setup the Jenkins home directory

    The Jenkins master stores its data in a local directory. Kubernetes will decide on which worker node the Jenkins-master will be deployed. Each of the worker nodes needs to use the same data. The data will be shared using NFS. Assuming you have setup a NFS server execute the following command on each of the worker nodes to create the Jenkins home directory at its default location.

    mkdir /var/jenkins_home

    In a multi worker node deployment add the NFS mount to /etc/fstab as shown in the fragment below

    <ip-address>:/var/data/jenkins /var/jenkins_home/ nfs rsize=8192 and wsize=8192,noexec,nosuid

    Where

    • <ip-address> is the ip address of the NFS server
    • /var/data/jenkins is the directory of the share on the NFS server

     

    To activate the NFS share reboot the worker node. After reboot check the share to work by putting/deleting a file in de share from the different nodes.

  2. Allocate persistent storage

    Create the file jenkins-persistent-storage.yaml with the following content:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
    name: jenkins-storage
    spec:
    accessModes:
    - ReadWriteOnce
    capacity:
    storage: 8Gi
    hostPath:
    path: /var/jenkins_home

    The 8Gi is the storage requirement for the Jenkins chart to be applied later. Apply the persistent storage definition using:

    kubectl apply -f jenkins-persistent-storage.yaml
  3. Create a namespace

    This step is optional. We have the tooling in a separate namespace.

    In the ICP UI goto Manage > Namespaces > Create Namespace and give your namespace a name (in this recipe we use inno-tools as the namespace).

  4. Add the Google helm chart repository

    In the ICP UI goto Manage > Helm Repositories > Add Repository and set

    Attribute Value
    Name Google
    URL https://kubernetes-charts.storage.googleapis.com

     
    and click Add

    The list of Helm repositories is shown. Click on the Sync repository button to populate the catalog.

  5. Create image policy for docker hub images

    The Google Jenkins chart refers to images on docker hub. You need the allow the use of these images.

    In the ICP UI goto Manage > Resource Security > Image Policies > Create Image Policy and set:

    Attribute Value
    Name docker.io
    Scope Namespace
    Namespace select your namespace (ours inno-tools)

     
    and click on Add Registry and set

    Attribute Value
    Registry URL docker.io/**
    VA scan policy Do not install (the default value)

     
    Click Add to add the Registry

    Click Add to add the Image policy

  6. Install Jenkins

    In the ICP UI goto Catalog (in the top navigation bar) and type jenkins in the search field. You will see the IBM-chart and the Google chart for Jenkins. The Google chart installs the latest Jenkins LTS release, which is at the time of writing version 2.138.

    Select the Jenkins Google chart for installation. The description of the chart is shown, click on the Configure button and set:

    Attribute Value
    Helm release name jenkins (or set to your naming conventions)
    Target namespace (in our setup inno-tools)

     
    See the following screenshot for the filled in fields.

    jenkins-config-release

    The default parameters will not give you a running Jenkins. Please open the All parameter section. The defaults for the first fields will do and are shown in the following screenshot:

    jenkins-config-first-fields

    The values in the next section need to be changed (the default values are shown)

    jenkins-config-second-fields

    The initial request of memory is insufficient. Even With a request of 1Gi and limit of 2Gi Jenkins will go into a restart cycle due to Out of Memory, so set

    Attribute Value
    resources.requests.memory 2Gi
    resources.limits.memory 4Gi
    ServiceType NodePort *)

     
    *) NodePort:  At the end of this recipe the Ingress resource will be configured to handle the incoming traffic. Only after enabling the Ingress controller the ServiceType can be set to ClusterIP.

    The following screenshot shows the updated section.

    jenkins-config-resources You can scroll down to the Persistence section as shown in the following screenshot.

    jenkins-config-persistent-storage

    No changes are required, just check that Size matches the allocated size of the persistent storage created earlier.

    You are now ready to install the chart, click Install and see the following message.

    jenkins-config-installing

    Click on the View Helm Release button and you will see the details of the helm release as shown in the next screenshot.

    jenkins-release-header

  7. Extract the Jenkins admin password

    During the installation a random password is generated for the Jenkins admin account. Assuming you are still on the Jenkins helm release page, goto the bottom of the page. Point 1 shows the command to retrieve the Jenkins admin password. Run the command to retrieve the generated admin password. Depending on your setup you may perform the other steps as well, but these steps are not necessary.

    You can reach the Jenkins administration page using the button Lauch at the top of the screen, but for now you need to wait a couple of minutes to have ICP complete the deployment.

  8. Wait for deployment to complete

    Assuming you are still on the Jenkins helm release page, scroll to the middle of the page to view the status. Note that the status on the page will refresh without refreshing the page itself. The following screenshot shows the deployment status.

    jenkins-release-status

    The number of restarts of the pod shall stay at 0. If the number of restarts increases then there is something wrong within the deployments, see the next step for tips for debugging.

  9. Problem determination

    Following the steps in this recipe, you should not run into an issue with the deployment. In case you do into an issue, use the following steps to get more info on what happened during the deployment.

    The logs will be not of help in resolving your issue, the info comes from Kubernetes:

    Run

    kubectl get pods –all-namespaces

    From the output, copy the pod identifier of the Jenkins pod (something like jenkins-85576df4f6-mv7tf) to the clipboard. Use the following command to get the details of the pod (note use your copied pod identifier and your own namespace).

    kubectl describe pod jenkins-85576df4f6-mv7tf -n inno-tools

    In case of insufficient memory you will see:

    Last State: Terminated
    Reason: OOMKilled
    Exit Code: 137
  10. Fix Jenkins reverse proxy message

    Unfortunately Jenkins does not have all its URLs relative to each other. So a bit of work is required from your side.

    Assuming you are still on the Jenkins helm release page, click the Launch button at the top right corner. On the Jenkins login page enter the user name (=admin) and the password retrieved in step “Extract the Jenkins admin password”.

    After logging in, goto Manage Jenkins. You will see the following message (might take a couple of seconds before shown):

    jenkins-admin-reverse-proxy-message

    Open the ‘Configure Jenkins’ page and scroll to the section with the Jenkins URL as shown in the following screenshot.

    jenkins-admin-location

    Replace the URL with the URL from your browser, like http://10.10.80.12:30449, the ip address is the ip-address of the master node, the port is the node port. Save the updated configuration and the message should disappear.

    Jenkins should work proper now, but is not reachable through the Ingress controller yet.

  11. Enable Ingress for Jenkins

    So far Jenkins is on http and using a NodePort. This is OK for the first setup, but not for having a production development environment. The goal is to have all incoming traffic running through the Ingress controller. This step is broken down in a number of configuration changes which all need to be completed. While making the changes Jenkins might become in anone running or working state.

     

    Change the Jenkins URL

    Login into Jenkins as admin and select Manage Jenkins > Configure System. Change the location URL as shown in the following screenshot.

    jenkins-ingress-location

    Where https://10.10.80.25:8443 is the entry point for your ICP cluster.

     

    Add the Ingress descriptor

    Create the file jenkins-ingress.yaml with the following content (change the namespace, service name if required).

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    annotations:
    kubernetes.io/ingress.class: ibm-icp-management
    name: jenkins
    namespace: inno-tools
    spec:
    rules:
    - http:
    paths:
    - backend:
    serviceName: jenkins
    servicePort: 8080
    path: /jenkins

    And apply the yaml by running

    kubectl apply -f jenkins-ingress.yaml

     

    Update the Jenkins deployment descriptor

    Goto the ICP UI to Workloads > Deployments as shown in the following screenshot.

    jenkins-ingress-deployments

    and open the menu next to the Launch link and select the Edit.

    Make the following changes in the deployment descriptor.

    In the container section find the line “name”: “JENKINS_OPTS”, add a comma to that line and add the line

    "value": "--prefix=/jenkins"

    The prefix tells Jenkins to prefix all URL it generate with /jenkins. The updated deployment descriptor is shown in the following screenshot.

    jenkins-ingress-jenkins_opts

    In the section “livenessProbe” change the path from “/login” to

    "/jenkins/login"

    as shown in the following screenshot.

    jenkins-ingress-livenessprobe

    Make the same change in the readinessProbe section as shown in the following screenshot.

    jenkins-ingress-readinessprobe

    Submit the updated deployment configuration. The status of the Jenkins deployment will change, the current, ready and available will be cleared. After a few minutes the deployment should become available.

     

    Open Jenkins

    In your browser take your ICP URL and add /jenkins, example url is https://10.10.80.25:8443/jenkins . All the effort should have resulted the Jenkins login page.

    jenkins-login-page

     

    You are set to further configure Jenkins and build your pipelines.

  12. Update the service type to ClusterIP

    Depending on your deployment/organisation you might not want to have Jenkins exposed over the nodeport. Firstly, HTTP is the default protocol, meaning that all data including the passwords will be sent unencrypted. Secondly, the goal of the cluster is to hide internals, and in a multi work node environment, the workload may move to another worker node in the cluster.

    In the ICP UI goto Network Access > Services. The services list is shown, see the following screenshot.

    jenkins-service-list

    Next to the created column, goto the menu and select the Edit menu item. The content is shown in the following screen shot.

    jenkins-service-before

    Remove the line with the NodePort and change the service type from NodePort to ClusterIP as shown in the following screenshot.

    jenkins-service-after

    Click on Submit. As result Jenkins is no longer available on the NodePort and Jenkins is only available through the Ingress controller of the cluster.

     

     

  13. Run a Jenkinsfile

    The last step to validate the installation is to run a Jenkinsfile. As running on Kubernetes the compiles and test are expected to be within pods. The pods can create a test environment including your application and a database server, this to isolate the test completely.

     

    Update number of executors

    After installation the number of executors is set to zero. In Jenkins goto Manage Jenkins > Configure System and set

    Attribute Value
    # of executors 1
    SCM GIT

    Authorize Jenkins in the Jenkins namespace

    Run

    kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin
    --user=admin --user=kubelet --group=system:serviceaccounts:inno-tools

    to allow Jenkins to create and monitor pods in its own namespace. Note you need to make one statement out of the two lines in the code block.

     

    Update the Jenkins Kubernetes configuration

    In Jenkins goto Manage Jenkins > Configure System – Cloud – Kubernetes and set

    Attribute Value
    Jenkins URL http://jenkins:8080/jenkins (as jenkins is no longer located at /)

     Scroll down to the Kubernetes Pod Template section and set

    Attribute Value
    Namespace inno-tools

     

    Create Jenkinsfile in your github

    Create a Jenkinsfile in your github project with the following content

    def label = "waiting-game-${UUID.randomUUID().toString()}"

    podTemplate(label: label,
    yaml: """
    apiVersion: v1
    kind: Pod
    metadata:
    namespace: inno-tools
    spec:
    containers:
    - name: ubuntu
    image: ubuntu
    tty: true
    """
    ) {
    node(label){
    stage('Run'){
    container('ubuntu') {
    echo 'waiting game'
    sleep 5
    echo 'bye'
    }
    }
    }
    }

     

    Create pipeline

    In Jenkins goto New Item, give your pipeline a name, select pipeline and click OK. In the definition goto the Pipeline section and set.

    Attribute Value
    Definition Pipeline script from SCM
    SCM GIT

    And complete the definition to access your GIT project.

    In Jenkins goto your pipeline and watch the console ouput. This should look like:

    Started by user admin
    Obtained Jenkinsfile from git https:<your github project>
    Running in Durability level: MAX_SURVIVABILITY
    [Pipeline] podTemplate
    [Pipeline] {
    [Pipeline] node

    Still waiting to schedule task
    ‘Jenkins’ doesn’t have label ‘waiting-game-48ff231c-c801-4e87-adca-2fb04a89f74f’

    Agent jenkins-slave-vm2pb-jg82p is provisioned from template Kubernetes Pod Template
    Agent specification [Kubernetes Pod Template] (waiting-game-48ff231c-c801-4e87-adca-2fb04a89f74f):
    yaml:

    apiVersion: v1
    kind: Pod
    metadata:
    namespace: inno-tools
    spec:
    containers:
    - name: ubuntu
    image: ubuntu
    tty: true


    Running on jenkins-slave-vm2pb-jg82p in /home/jenkins/workspace/jsone
    [Pipeline] {
    [Pipeline] stage
    [Pipeline] { (Run)
    [Pipeline] container
    [Pipeline] {
    [Pipeline] echo
    waiting game
    [Pipeline] sleep
    Sleeping for 5 sec

    [Pipeline] echo
    bye
    [Pipeline] }
    [Pipeline] // container
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] // node
    [Pipeline] }
    [Pipeline] // podTemplate
    [Pipeline] End of Pipeline
    Finished: SUCCESS

Join The Discussion