Digital Developer Conference: Hybrid Cloud 2021. On Sep 21, gain free hybrid cloud skills from experts and partners. Register now

Configure Jenkins in high availability mode on IBM Cloud

Introduction

It is critical for components of a DevOps toolchain to maintain high availability (HA) for continuous delivery to be successful. Infrastructure as code (IaC) plays a crucial role by eliminating manual processes and eliminating the slack in the process. This code-based approach makes it easier to get more done in less time: There’s no need to wait on IT administrators to manually complete a task before moving on to the next one.

Jenkins is an open-source Java-based application that works with plug-ins. It is mostly used as a continuous integration (CI) engine. It also facilitates continuous build and deployment and infrastructure deployment. Jenkins is one of the most popular CI engines, and because it is a free and open-source, Jenkins is one of the most commonly used CI engine. Because multiple teams use Jenkins, it is critical that is remains highly available.

This tutorial introduces two methods for configuring Jenkins in HA mode by using IaC on IBM Cloud:

  • Method 1: Configure Jenkins in HA mode on IBM Cloud using Terraform and Ansible scripts
  • Method 2: Use schematics templates on IBM Cloud for a template-based approach

You can also adapt these methods to other cloud platforms (for example, AWS) while configuring your DevOps toolchain.

Prerequisites

To follow along, you need:

  • An IBM Cloud account
  • Docker
  • Jenkins
  • A GitHub repository
  • Basic knowledge of software load balancing (NGINX or HAProxy)
  • Basic knowledge of Terraform and Ansible
  • Basic knowledge of Continuous Integration/Continuous Deployment (CI/CD)

Estimated time

Completing this tutorial should take about 30 minutes.

Provision an NFS store on IBM Cloud

The following figure shows a high-level diagram of the Jenkins architecture implementation using IBM Cloud:

Available file stores

In the image:

  1. The user sends traffic to the software load balancer.
  2. The software load balancer directs the traffic to the Jenkins controller. If the controller is down, the traffic is redirected to a Jenkins backup server.

Back-end storage is important for any application to be highly available. Both of the Jenkins nodes share the same storage for Jenkins, which is an NFS store, which can share storage across multiple nodes. Let’s start by provisioning the back-end storage for Jenkins:

  1. From the IBM Cloud navigation menu, click Classic Infrastructure > File Storage.
  2. Click Order File Storage.

Enter the required details in the portal and provision the file storage. You need to select the cloud region of the file store, the size of the file store, and the billing method (monthly or hourly). Here, we have selected an endurance general type of storage. Once you provision the file store, you can view the file stores available on the system.

Available file stores

Provision a load balancer server and 2 Jenkins servers on IBM Cloud

You can provision the servers using any of the following options:

  • IBM Cloud CLI
  • IBM SoftLayer Portal
  • Terraform

This tutorial uses the following Terraform scripts to provision 3 servers: 1 for load balancing and 2 for Jenkins.

The main.tf script:

 data "ibm_compute_image_template" "centos" {
  name = var.os
}
resource "ibm_compute_vm_instance" "virtualguest" {
  count                = var.vm_count
  hostname             = "haproxypoc${count.index + 1}"
  domain               = "abc.com"
  image_id             = data.ibm_compute_image_template.centos.id
  datacenter           = var.datacenter
  private_vlan_id      = var.vlan_id
  network_speed        = 100
  ssh_key_ids          = ["11111"]
  hourly_billing       = true
  private_network_only = true
  flavor_key_name      = var.vm_flavor
  local_disk           = false
}

The variables.tf script:

variable "vip" {
  default = "10.114.128.14"
}
variable "vm_count" {
  default = "2"
}
variable "vm_flavor" {
  default = "B1_2X4X100"
}
variable "vlan_id" {
  default = "9999999"
}
variable "datacenter" {
  default = "tor01"
}
variable "os" {
  default = "centos"
}
variable "http_port" {
  default = {
    "0" = "80" #haproxy
  }
}

The provider.tf script:

variable "iaas_classic_username" {
  description = "Enter the user name to access IBM Cloud classic infrastructure. "
}

variable "iaas_classic_api_key" {
  description = "Enter the API key to access IBM Cloud classic infrastructure. "
}

variable "ibmcloud_api_key" {
  description = "Enter your IBM Cloud API Key "
}

provider "ibm" {
  iaas_classic_username = var.iaas_classic_username
  iaas_classic_api_key  = var.iaas_classic_api_key
  ibmcloud_api_key      = var.ibmcloud_api_key
}

The versions.tf script:

terraform {
  required_version = ">= 0.12"
}

Configure HAProxy (load balancing server)

You use an Ansible role to run and configure HAPproxy on each of the servers. The main module is playbook_haproxy.yml.

The following code shows the Ansible playbook for configuring the load balancing server:

---
- hosts: all
  remote_user: root
  become: yes

  roles:
    - haproxy

  tasks:
  - name: shutdown haproxy cluster
    shell: /usr/local/bin/docker-compose -f /app/haproxy/docker-compose.yml down

  - name: start haproxy
    shell: /usr/local/bin/docker-compose -f /app/haproxy/docker-compose.yml up -d haproxy

You can find more details about HAProxy in the configJenkins GitHub repo.

The configuration file for HAProxy is shown below. The HAProxy accepts traffic on port 80 and forwards it to port 8080 on the Jenkins server.

defaults
  log global
  maxconn 2000
  mode http
  option redispatch
  option forwardfor
  option http-server-close
  retries 3
  timeout http-request 10s
  timeout queue 1m
  timeout connect 10s
  timeout client 1m
  timeout server 1m
  timeout check 10s

frontend ft_jenkins
 bind *:80
 acl host_cicdprod hdr(host) -i cicdprod.abc.com
 use_backend bk_jenkins if host_cicdprod
 default_backend bk_jenkins
 reqadd X-Forwarded-Proto:\ http

backend bk_jenkins
  server jenkins1 {{ cicdhostprod1.stdout }}:8080 check
  server jenkins2 {{ cicdhostprod2.stdout }}:8080 check backup

Configure Jenkins

You configure Jenkins using Ansible. The main module is playbook_jenkins.yml. It uses Docker Compose to spin up Jenkins.

The following code shows the docker-compose file:

version: '2'
services:

  jenkins:
    image: jenkins:lts
    container_name: jenkins
    ports:
      - "8080:8080"
      - "50000:50000"
    restart: always
    environment:
      - TZ=America/Toronto
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

    network_mode: "host"

Schedule a cron job to sync files between the Jenkins servers

Jenkins runs on 2 servers. You need to sync up the logs across both of the nodes. To do this, you use a shell script on both the nodes and use cron to schedule the job every minute.

The following code shows the cron job. The cron is added as a part of playbook_jenkins.yml.

#!/bin/bash
crumb_id=$(curl -s 'http://localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)',-u admin:admin)
curl -s -XPOST 'http://localhost:8080/reload' -u admin:admin -H "$crumb_id"

The script calls the Jenkins reload API to reload the Jenkins configuration every minute.

Automate using IBM Cloud Schematics and DevOps toolchain

This section automates the provisioning and configuration of Jenkins in HA using the IBM Cloud toolchain.

Terraform

  1. Upload main.tf, provider.tf, variables.tf, and versions.tf to your GitHub repo.
  2. From the IBM Cloud console, click > Schematics > Create a Schematics Workspace. Enter a unique workspace name, and then click Create.
  3. Enter your GitHub repository URL and click Save template information.
  4. Configure the variables as described in the table to authenticate the API keys and user credentials and save the changes. These are the same variables that were declared in the variables.tf and provider.tf files. The values mentioned in the script are populated. The cloud API keys are not mentioned here for confidentiality reasons.

    Variable configuration

  5. Click Enable continuous delivery to create the DevOps toolchain.

  6. Enter the following details for the toolchain. This creates the toolchain repository on Git.

    • Toolchain name: Name of the toolchain to be created
    • Region: Region where the toolchain must be created
    • Resource group: Select the resource group
    • Repository type: New
    • IBM Cloud API key: IAM key for IBM Cloud

      Create the toolchain repository

  7. From the IBM Cloud console navigation menu, click Schematics > Workspaces. Select the location from the drop-down. Click on your workspace.

    Workspace

  8. Click Generate plan.

    Generate and apply plan

  9. Click View log.

    View the log

  10. If there are no errors in the log, click Apply plan.

  11. From the navigation menu, click DevOps to view the toolchain.

    View the DevOps toolchain

Ansible

Ansible is an open-source configuration language. This is one of the most simple configuration management tools and is agentless.

  1. Upload the Ansible playbook to GitHub.

     ---
     - hosts: all
       remote_user: root
       become: yes
    
       roles:
         - haproxy
    
       tasks:
       - name: shutdown haproxy cluster
         shell: /usr/local/bin/docker-compose -f /app/haproxy/docker-compose.yml down
    
       - name: start haproxy
         shell: /usr/local/bin/docker-compose -f /app/haproxy/docker-compose.yml up -d haproxy
    

You create two actions: one for Jenkins (called hajenkins) and the other for nginx (called loadbalancer).

  1. From the IBM Cloud console navigation menu, click Schematics.
  2. In the Configuration management box, click Create action.
  3. Fill in the form to create an action:
  4. Enter the GitHub URL and click Retrieve playbooks.

    Retrieve playbook

  5. Click the Playbook drop-down list to select the playbook.

  6. Click the Verbosity drop-down list to select the level of logging.
  7. In the Key field, enter the extra variables (variables that are used as input with -e option) and click Save.

    Playbook variables

  8. Click Create Inventory to create the inventory file that will have the hosts in IBM Cloud Classic Infrastructure. You can also enter the bastion host IP and private key, which the playbook uses to log into the hosts.

  9. Click Run Action to execute the playbook.

    Playbook variables

  10. View the Ansible log:

    Ansible log

  11. Repeat these steps to create an action for Jenkins.

Summary

Jenkins is a CI/CD orchestrator and is used by the developers, testers, and DevOps teams. In this tutorial, you learned how to configure Jenkins in high availability mode on IBM Cloud. The method described here is cloud-agonistic (in other words, it can be executed on any cloud platform). This uses IaC and so the configuration can be maintained on GitHub or a Git-based repository.