Following my previous article where I showed how to set up IBM Integration Bus running on SoftLayer servers, I have progressed on to automating the installation and configuration of IBM Integration Bus.  I chose to focus on the popular Chef framework for this purpose.  The Chef framework allows you to manage a set of “nodes” on which software is installed, and a set of “cookbooks” which contain “recipes” which do the software installation.  The recipes that are used in this article are available on the Open Technologies for Integration cookbooks github site. In this article I describe using SoftLayer servers to host components of the solution, however using Chef to automate installation is equally valid for other hosted infrastructure providers, or within an enterprise using on-premise infrastructure.

In this article I’ll describe how to set up a Chef server, workstation and nodes so that the IBM Integration Bus installation and configuration on those nodes can be automated.  The Chef server and nodes each run on SoftLayer Cloud Compute Instances (CCIs), and the Workstation is my laptop.  The following diagram shows these components and the main interactions between them:

This article explains how the three components are set up as follows:

  1. A SoftLayer CCI is created and the Chef server is installed on that instance.
  2. The local Workstation has the Knife software and cookbooks installed.  Knife is the administrator’s command line tool for interacting with the Chef Server.
  3. A SoftLayer CCI is created and Chef recipes are used to install IBM Integration Bus on that instance.
  4. The installation is verified using a remote desktop connection, web admin user interface or an MQ connection from a local Integration toolkit

Note that I chose to use SoftLayer CCIs for both the Chef server and node.  You could install the Chef server locally and use SoftLayer’s 1 month free offer for the node, in which case you must ensure that the SoftLayer CCI is able to connect to your Chef server.

Installing the Chef server

The Chef server manages cookbooks, a set of nodes, the recipes that have been installed on those nodes and the nodes’ current state. The Chef server can run anywhere that can be contacted by the nodes; in this article I create a SoftLayer CCI and install the Chef server software on it, along with an FTP server that provides access to the IBM Integration Bus installation package.

Creating the SoftLayer CCI for the Chef server

To create a SoftLayer CCI for the Chef server, starting from the SoftLayer Management Web Page:

  1. From the Sales tab, select “add monthly computing instance”.
  2. On the first page, select the data center, processor, RAM and networking.  For the Chef Server, I chose 2 cores with 4GB RAM and default settings for networking.
  3. On the next page, select storage choices. I chose a 25GB local disk and took the default settings for everything else.
  4. Select the O/S and software. I chose “Red Hat Enterprise Linux 6 – Minimal Install (64 bit) (1-4 Core)” with no additional software.  You can select any operating system supported by the Chef server at this point.
  5. You next have the option to select management services, firewall etc. I just accepted the defaults.
  6. On the final page, specify the configuration for the server: SSH keys, host and domain name, and an optional provisioning script with associated metadata which is the basis for automated software install and configuration.

Once you have selected the items in the steps, it should take around an hour for the server to be created and set up for you.

Setting up the Chef server

Once the SoftLayer server is provisioned, you can install the Chef server.  You’ll need to SSH as root into the server using the root password provided on the CCI properties page and perform the following commands (note that the wget command is shown split over two lines but should be entered as a single command):

wget https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.8-1.el6.x86_64.rpm
sudo rpm –ivh chef-server-11.0.8-1.el6.x86_86.rpm
echo erchef[\'s3_url_ttl\']=3600 >> /etc/chef-server/chef-server.rb
sudo chef-server-ctl reconfigure

The chef-server setting is required to allow recipes to run for more than 15 minutes without security tokens expiring; the value 3600 allows for up to one hour.

In the same session you can then verify the installation by using the supplied test case command:

sudo chef-server-ctl test

After you complete verification, you can log into the Chef server web user interface by pointing your browser at the SoftLayer server’s IP: https://<Chef server IP address>:443/  Log in to the server with the default admin ID and password, which is shown on the initial web page, and then change the password as prompted.

You’ll see that initially the Chef server has no knowledge of recipes, nodes, etc.  These will be set up in the next step from your workstation using the “knife” command.

For more information, this blog article provides an overview of installing the Chef server: http://www.opscode.com/blog/2013/03/11/chef-11-server-up-and-running/

Enabling IBM Integration Bus v9 download from the Chef server

In order to allow IBM Integration Bus to be installed using Chef, it is necessary to make the install package directly downloadable to the servers that Chef is managing.  There are several ways to achieve this, including the use of an FTP or HTTP server or shared file system.  For this article I chose to make the package available via anonymous FTP.  First you need to download the IBM Integration Bus Developer Edition.

I used the single Linux package and installed everything on the SoftLayer server.  When downloading the Developer Edition, select “IBM Integration Bus for Developers (single package download), and click continue. Provide the requested details, and then select the Linux package as shown below:

You can also download the Linux runtime package and Windows or Linux Toolkit package if you want to run the Toolkit locally and the runtime on the SoftLayer server, but the approach described in this article would need to be modified in that case.

As with the previous article I’m using vsftpd as the FTP daemon with a couple of modifications to the default configuration.  SSH as root to the Chef server and execute the following commands:

yum install vsftpd
vi /etc/vsftpd/vsftpd.conf     <- comment out anonymous_enable
vi /etc/vsftpd/ftpusers        <- comment out root
vi /etc/vsftpd/user_list       <- comment out root
service vsftpd start
chkconfig vsftpd on

Once those commands have completed, upload the IBM Integration Bus install package to the FTP server from the machine on which the package was downloaded by using the following commands:

ftp <SoftLayer Chef server>
<Login as root>
cd /var/ftp/pub
mkdir iib
cd iib
bin
put 9.0.0-IIB-LINUXX64-DEVELOPER.tar.gz
exit

Note that because the full install package is 3GB it takes approximately two hours to transfer, but this is a one-time operation. Once the file transfer has completed, you can revert the changes to the ftpusers and user_list files to remove root ftp access as it won’t be needed again.

Set up the local workstation

In Chef terminology, the “Administrator’s Workstation” means the computer from which you will be managing cookbooks and nodes.  The two tools that are needed on the workstation are GIT for managing files and Knife for creating Chef resources and interacting with the Chef server.

The following steps are described in this section:

  1. Ensure that the Chef server can be contacted from the workstation
  2. Install Knife on the workstation
  3. Download the cookbook and recipes
  4. Create an IBM Integration Bus role

Once these are complete you will be ready to install IBM Integration Bus for Developers on any node.

Ensure that the Chef server can be contacted from the Workstation

The first step is to ensure that the Chef server can be contacted from your workstation.  This requires the Chef server hostname to be resolvable from your workstation. When I created the SoftLayer servers I chose host names in the softlayer.com domain, which are not automatically published to DNS. Setting up DNS correctly would normally be done as part of the overall networking setup on SoftLayer with your own domain so this step would not be required in that case.

In order to allow my workstation to use the SoftLayer server host name I added it to the “\etc\hosts” file. On Windows, use an elevated command prompt (Start -> All Programs -> Accessories, right click “Command Prompt” and select “Run as Administrator”), then edit the file c:\windows\system32\drivers\etc\hosts to add the following line, substituting in the IP address and host name of the Chef server:

<Chef server IP address> <Chef server host name> # Chef server

The end of the hosts file should then look something like the following:

# localhost name resolution is handled within DNS itself.
      127.0.0.1       localhost
#     ::1             localhost
10.27.5.1 iib-chef.softlayer.com # Chef server

You can verify this using the “ping” command to contact the chef server from a command prompt on the Workstation:

ping iib-chef.softlayer.com
Pinging iib-chef.softlayer.com [10.27.5.1] with 32 bytes of data:
Reply from 10.27.5.1: bytes=32 time=180ms TTL=46
Reply from 10.27.5.1: bytes=32 time=162ms TTL=46

Install Knife on the workstation

Next install Knife on your workstation following the linked article applying the comments below.  I would recommend downloading the “.pem files” mentioned in step 3 before you go through this process:

  1. In the “Clone the Chef Repo” step I chose to put the chef-repo as a subdirectory of a directory where I keep my work projects (c:\work\chef).
  2. In the “Create the .chef directory” step I created the .chef directory my Windows user directory (c:\users\hldswrth\.chef)
  3. In the “Get the .pem files” step I used WinSCP to transfer the chef-validator.pem and admin.pem from the SoftLayer Server’s /etc/chef-server directory to my local .chef directory.  Note that you need to do this before running the “knife configure” command.
  4. To create the knife.rb file I used the knife command “knife configure –i” as shown in the screenshot below:

Once this step is complete and you have completed the step to verify the Knife install, you can proceed to add cookbooks, recipes and roles to the Chef server, ready to set up your first Chef node.

Create Cookbooks and Recipes

Creation of cookbooks is done using the “knife cookbook” command.  To create a cookbook from scratch you would use the command:

knife cookbook create <name, e.g. ibm_integration_bus>

The Open Technologies for Integration (“OT4I”) Cookbooks github site has an ibm_integration_bus cookbook for installing IBM Integration Bus which provides 2 recipes:

  • The “default” recipe installs IBM Integration Bus v9 and its dependencies.
  • The “runtime” recipe installs only the runtime components of IBM Integration Bus. You will need a separate install of the toolkit and explorer components to manage and deploy the runtime.

This github site also provides a vnc cookbook for setting up a remote desktop connection which provides one recipe:

  • The “default” recipe sets up a VNC server to allow you to make a remote desktop connection into the node for the purposes of running the Integration toolkit.  This recipe also installs the Firefox web browser and LXDE and sets the LXDE desktop to start when the VNC server is started.  Note that this recipe only installs non-IBM software and is provided as-is.

Use the following commands on the workstation to clone these cookbooks into your local chef repository:

cd <chef-repo directory>\cookbooks
git clone https://github.com/ot4i-cookbooks/ibm_integration_bus
git clone https://github.com/ot4i-cookbooks/vnc

This should result in the following files (at least) being downloaded:

ibm_integration_bus\CHANGELOG.md
ibm_integration_bus\FAQ.md
ibm_integration_bus\metadata.rb
ibm_integration_bus\README.md
ibm_integration_bus\attributes\default.rb
ibm_integration_bus\recipes\default.rb
ibm_integration_bus\recipes\runtime.rb
ibm_integration_bus\templates\default\IIBService.erb

vnc\CHANGELOG.md
vnc\FAQ.md
vnc\metadata.rb
vnc\README.md
vnc\attributes\default.rb
vnc\recipes\default.rb
vnc\templates\default\VNCService.erb

Once cloned into your local chef-repo, upload these cookbooks to the Chef server using the following commands:

knife cookbook upload ibm_integration_bus
knife cookbook upload vnc

You can then use the Chef server web interface (at IP: https://<Chef server IP address>:443/ ) to view the cookbooks on the server:

The recipes include a number of “attributes” which allow specific values to be provided each time they are run.  In particular, the name and location of the installation package are provided as attributes so that they can be specified in your environment.  The next step is to specify the recipes and attribute values to be used for a particular installation “role”.

Creating an IBM Integration Bus Role

In order to install IBM Integration Bus using these recipes, you must tell the Chef server which recipes to install on each node. You can specify this each time you run the Chef client on a node, or provide the information via a definition known as a “role”.  The role identifies a list of recipes to be run on a node, along with a supplied set of default and override attributes.  This allows the same recipes to be customized when running on nodes in different roles.

For a default installation and configuration of IBM Integration Bus, you can create a role called “iib_default”.  This is done using the “knife role” command on the workstation.  When creating the role, the knife command needs to launch an editor so that you can tailor the role definition.  On Windows I use Notepad, as follows:

set EDITOR=notepad
knife role create iib_default

This will launch Notepad with a default role definition in JSON format.  First, modify the “override_attributes” to point to the location of the install image, and specify the user name that will be used to create runtime objects and run the VNC server.  Second, modify the “run_list”, which specifies the recipes to be run on the node.  Edit those two elements in the file so that they look like the text below, highlighted in blue. For this example, you should set the “account_username” attribute override to the name of the user under which you will be running the Integration Toolkit and the VNC server, and both should have the same value.  This user will be created for you on the node:

{
  "name": "iib_default",
  "description": "",
  "json_class": "Chef::Role",
  "default_attributes": {
  },
  "override_attributes": {    
    "ibm_integration_bus": {
      "package_site_url": "ftp://iib-chef.softlayer.com/pub/iib",
      "account_username": "hldswrth"
    },
    "vnc": {
      "account_username": "hldswrth"
    }
  },
  "chef_type": "role",  
  "run_list": [
    "recipe[ibm_integration_bus]",
    "recipe[vnc]"
  ],
  "env_run_lists": {
  }
}

Save the role definition and exit the editor.  If you entered the text correctly, Knife will upload the role definition to the Chef server automatically, otherwise it will indicate where the error is, and you will nee to re-run the “knife role create” command.  You can modify the role definition using the “knife role edit” command if needed.

Use Chef to Automate Installation of IBM Integration Bus

Now that all the necessary setup has been done to define the Chef role and make the IBM Integration Bus install package available on the Chef server, nodes can be added to the Chef server and have IBM Integration Bus automatically installed and configured.

In this article I am using SoftLayer CCIs for nodes so the first step is to request an additional Cloud Computing Instance for each node on which you wish to run IBM Integration Bus. This is similar to the Chef server, using appropriate selections for number of processors, memory and disk (minimum 4 cores, 4GB memory and 25GB disk).  For this article I created the node CCI using “Ubuntu Linux 12.04 LTS Precise Pangolin – Minimal Install (64 bit)” as the “ibm_integration_bus” cookbook recipes at time of writing currently run on Ubuntu.

Once the CCI for a node is provisioned, you need to perform two steps to ensure the recipes can be run on that CCI.  The first step is to ensure that the Chef server’s host name is resolvable. The second is to ensure that the SSH sessions used by the knife command don’t time out.  To achieve both of these, start an SSH session as root to the node CCI and enter the following commands.  Make sure you use “>>” where indicated so that the information is appended to the files:

cp /etc/hosts hosts.bak
cp /etc/ssh/sshd_config sshd_config.bak
echo <Chef server IP address> <Chef server host name> >> /etc/hosts
echo ClientAliveInterval 60 >> /etc/ssh/sshd_config
echo ClientAliveCountMax 60 >> /etc/ssh/sshd_config
service ssh restart

If you don’t do this, you might see the following error indicating that the node cannot resolve the Chef server’s host name:

Error connecting to https://<Chef server host>//clients - getaddrinfo: Name or service not known

The SSH client connection settings on the server result in a keep-alive interaction every 60 seconds to avoid the session timing out if any step in the recipes takes a significant amount of time.

Note that this step could be automated by specifying a provisioning script when you request the CCI.  The provisioning script runs these commands when the server is created.

Bootstrap the Node

Having completed the initial setup of the node, you can “bootstrap” the new node using Knife from the workstation using the following command. I used “iibnode1” as the node name for my first IBM Integration Bus node, the password is the root password for the CCI as shown on the CCI’s property page.

knife bootstrap <node IP address> 
   --node-name <node name> 
   --ssh-user root 
   --ssh-password <password> 
   --run-list role[iib_default]

Running this command causes the Chef client to be installed on the node, registers the node with the Chef server and downloads and runs all recipes for the defined role.  For me this took around 30 minutes.  Note that recipes are expected to be idempotent – you should be able to re-run them any number of times without getting a different result. However also note that if you do re-run the ibm_integration_bus recipe, it will reset the IBM integration bus runtime so any additional configuration or deployment you may have done after running the recipe will be discarded.

Verify the IBM Integration Bus installation on the Node

To verify the installation you can connect to the node and start the Integration Toolkit.

Using a remote desktop connection, with VNC

The vnc cookbook sets up a VNC server using TightVNC and desktop environment using LXDE, as the base Softlayer CCI does not include this capability.  Once the bootstrap process is complete, the VNC server is configured as a service.  Ideally this service would be started by the recipe so that you could connect immediately to the server using a VNC viewer, however I found it necessary to start the vncserver service manually through an SSH session in order to get the desktop to work properly.  Open an SSH session to the node, logging in as root, and enter the following commands:

su - <user name specified for the installation>
vncpasswd
<enter the password to be used for VNC connections and confirm it>
logout
service vncserver start

This will prompt you for a password to use to connect to the VNC server, and then start the VNC service. You should now be able to open a VNC connection to the node.  I used the TightVNC client on my workstation.  Note that the recipe by default sets up a non-secured connection to the server so it is recommended to use SSH to secure the VNC connection.  A web search for “VNC SSH tunnel” will provide various ways to achieve this. I found the instructions on this page useful, starting at the heading “Recommended Step – Secure your VNC server session with encryption”.

Start the VNC client and provide the node IP address, and VNC password (from the previous step) when prompted.  After entering the password you will see the LXDE desktop.  IBM Integration Toolkit is available as an option in the Applications menu:

Once the Integration toolkit has started, you can verify the installation by using the supplied pager sample, via the “Get Started” link provided on the “Welcome” page (Use the Help menu Welcome item to restore the Welcome page).  The ibm_integration_bus default recipe creates the Default Configuration automatically, so you don’t need to do that step.  Click on the “Verify your installation using the Pager samples” link to start verification.  If you see the following error, then you need to set up the web browser:

Navigate to Window->Preferences, General/Web Browser/New… add an entry for Firefox (as shown in this screenshot), then press OK.

You now have a fully installed, configured and running Developer environment for IBM Integration Bus.

Using the Integration Web Administration Page

You can connect directly to the node’s Integration Web Admin interface by opening a web browser on the workstation and entering the following URL in the address field:

http://<node IP address>:4414/

Once the page has loaded you should see the web admin console:

Using a local Integration Toolkit

If you have installed the Integration toolkit on your local machine you can connect that to the Integration Server on the node. If you are using a different user ID on your local workstation to the one you specified when the recipe was run, you need to define the local user ID on the node and add it to the mqm group.  This will enable you to connect from your local workstation to the Integration server running on the node via MQ.  To create the user on the node, SSH to the node as root and type the following commands:

sudo adduser <local user ID>
sudo usermod -G mqm -a <local user ID>

To enable a remote connection to the Integration node, SSH to the node using the local user ID just created (or that you used when you ran the recipe) then type the following commands:

. /opt/mqm/bin/setmqenv –s
runmqsc IB9QMGR
DEFINE CHANNEL(SYSTEM.ADMIN.SVRCONN) CHLTYPE(SVRCONN)
ALTER QMGR CHLAUTH(DISABLED)
refresh security
end

You will see the following output from the runmqsc command:

<local user name>@<node>:~$ runmqsc IB9QMGR
5724-H72 (C) Copyright IBM Corp. 1994, 2011.  ALL RIGHTS RESERVED.
Starting MQSC for queue manager IB9QMGR.
DEFINE CHANNEL(SYSTEM.ADMIN.SVRCONN) CHLTYPE(SVRCONN)
     1 : DEFINE CHANNEL(SYSTEM.ADMIN.SVRCONN) CHLTYPE(SVRCONN)
AMQ8014: WebSphere MQ channel created.
ALTER QMGR CHLAUTH(DISABLED)
     2 : ALTER QMGR CHLAUTH(DISABLED)
AMQ8005: WebSphere MQ queue manager changed.
refresh security
     3 : refresh security
AMQ8560: WebSphere MQ security cache refreshed.
end
     4 : end
3 MQSC commands read.
No commands have a syntax error.
All valid MQSC commands were processed.

You can then connect to the Integration Node from your local Integration toolkit:

  1. Log in to your workstation using the user name specified above
  2. Start Integration Toolkit on your workstation
  3. Right-click “Integration Nodes” in the “Integration Nodes” view, select “Connect to a remote Integration Node”, specifying the IP address, Queue Manager name (IB9QMGR) and port (2414).

Summary

This article shows how to set up a Chef server and use it to automate the installation of software – in this case IBM Integration Bus – on SoftLayer hosted servers.  The Chef setup and recipes described in this article are not SoftLayer specific, so you can use this approach on other hosted infrastructure providers, or within an enterprise using on-premise infrastructure.

1 comment on"How to use Chef Cookbooks to automate IBM Integration Bus Installs"

  1. Hi,

    I tried this few months ago and it looks there is no direct way to download the IBM Integration Bus v9. I am planning to try the above recipes to automate the IIB install on-premise infrastructure.

    Currently I can only download the IBM Integration Bus v10, so is it possible to use chef recipes to automate the IIB v10 install? If not is there work around for on-premise infrastructure to install using v10? At this point IIB version does not matter to me I am just doing a POC, so for my testing both version will work.

Join The Discussion

Your email address will not be published. Required fields are marked *