Introduction

This blog article describes how to provision IBM Integration Bus and IBM MQ within an Amazon Web Services(AWS) AMI instance and how to then invoke an application running in the instance that will use both products. The inclusion of IBM MQ provides the integration node with a full range of functions.

This article is an extension of the article Basic Deployment of IBM Integration Bus in Amazon Web Services which showed how to build and run an AMI consisting of IBM Integration Bus in AWS. As you may have come to this article directly, the whole process is outlined in this article so that it can stand as a complete guide on what to do.

This article assumes that you are familiar with key AWS concepts like EC2 and AMI along with JSON schema and bash shell scripting.

    To repeat the steps shown in this article in your own environment, you will need the following:

  • Amazon AWS account – you need to be able to deploy and manage EC2 resources
  • AWS client – this needs to be installed on your local machine. It provides a command line interface to AWS EC2 resources that allows you to create, deploy and manage AMI. See here
  • Packer – an open source tool which allows you to create virtual machine instances with a number of cloud providers, including AWS. See here. This needs to be installed on your local machine
  • JSON file editor – this can be any text editor but one that highlights and can validate JSON is a great help. This article used ATOM. See here

The sequence of steps needed to deploy IBM Integration Bus is as follows:

  1. Install the prerequisites and configure the AWS client as given in the documentation
  2. Build an AMI image that consists of a base Ubtuntu image on which IBM Integration Bus and IBM MQ are then installed
  3. Use AWS CloudFormation commands to provision one or more instances of the AMI created in step 2 above
  4. Verify the installation

Implementation

Step 1. Install the perquisites

As explained earlier, AWS client, Packer and a JSON editor need to be installed on your machine. This is well documented by those components and not described any further here. As part of the account set up you also need to create a key pair and ensure that they are deployed to any AWS regions that you will use. See here for details on how to do that.

Step 2. Build an AMI image

A base Ubuntu AMI image is used as a foundation on which to then install IBM Integration Bus. The Ubtunu AMI image is provided by Amazon. This article used an Ubuntu 14.04 LTS AMI that had an instance type of hvm:ebd-ssd in the eu-west-1 zone. See here for a list of AMI instances. At the time of writing the AMI used was AMI ami-2944b450.

    The AMI build step requires two files:

  1. A template that is in JSON format which is given as input to the Packer build process
  2. A bash script that will run as part of the build. This script specifies the processing that will be done to install the product and do any configuration. Note that it will install the IBM Integration Bus binaries but it will defer the licence acceptance and full configuration until the next step. The specific reason for this is explained later in the article but it is really to do with which resources are available at this time. IBM MQ will be installed and the licence accepted straightaway as the product behaves differently on licence acceptance

Template for Packer Build
The Template file used for packer is shown here:

{
  "provisioners": [{
    "type": "shell",
    "script": "iib-mq-install.sh",
    "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E {{ .Path }}",
    "environment_vars": [
      "MQ_URL=http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/messaging/mqadv/mqadv_dev90_linux_x86-64.tar.gz",
      "MQ_PACKAGES=\"MQSeriesRuntime-*.rpm MQSeriesServer-*.rpm MQSeriesMsg*.rpm MQSeriesJava*.rpm MQSeriesJRE*.rpm MQSeriesGSKit*.rpm\"",
      "IIB_VERSION=iib-10.0.0.10",
      "IIB_URL=http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/integration/10.0.0.10-IIB-LINUX64-DEVELOPER.tar.gz"
    ]
  }],
  "builders": [{
    "type": "amazon-ebs",
    "ami_name": "ibm-iib-mq-{{timestamp}}",
    "instance_type": "t2.micro",
    "region": "eu-west-1",
    "source_ami": "ami-2944b450",
    "ssh_username": "ubuntu",
    "ami_description": "IBM MQ V9.0, IBM Integration Bus V10.0.0.10 on Ubuntu 14.04"
  }]
}

The data value of script, within provisioners, is the name of the bash script that will run as part of the AMI build process. In this case it is iib-mq-install.sh. This name can be changed if needed but a script of the new name must be created and be available in the directory from which packer is invoked.

The environment_vars array specifies a number of environment variables that are specific to this build:

  • MQ_URL URL is the location from which the product binaries of IBM MQ are downloaded. In this article the developer edition of IBM MQ version 9.0.3 CD release is used for illustration purposes. This could easily be changed to use a local copy of the product binaries. You must have a valid license for the version of the product being used
  • MQ_PACKAGES is a list of the IBM MQ packages that will be installed
  • IIB_VERSION is the version of IBM Integration Bus that will be installed
  • IIB_URL is the location from which the product binaries of IBM Integration Bus are downloaded. In this article the developer edition of IBM Integration Bus version 10.0.0.10 is used for illustration purposes. This could easily be changed to use a local copy of the product binaries. You must have a valid license for the version of the product being used

In the builders section of the template:

  • ami_name specifies the name by which the new AMI will be known. Note the AMI id is the output of this build step and it will be generated by AWS
  • region specifies the name of the AWS region in which the AMI is to be built. This must match the region in which you want to deploy the running instances of the broker. The AWS client on the local machine must also be configured to use this region. See the AWS configure command for more details on how to do that
  • source_ami specifies the AMI id of the base Ubtuntu image. In this case was it was ami-2944b450 which is an Ubuntu 14.04 LTS AMI. This AMI must be available in the region in which the build is taking place. Note that not all AMI ids are available in all regions
  • ssh_username specifies the name of the user id that will be used when accessing the image using ssh. AWS AMIs use the Ubuntu id rather than root
  • ami_description is a description of the AMI for documentation purposes. In this case it is set as IBM MQ V9.0, IBM Integration Bus V10.0.0.10 on Ubuntu 14.04

Install Script

The install script, iib-mq-install.sh, consists of the following:

#!/bin/bash
# -*- mode: sh -*-
# (C) Copyright IBM Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Fail on error
set -e
export DEBIAN_FRONTEND=noninteractive
# Recommended: Update all packages to the latest level
apt-get update
apt-get upgrade -y
apt-get update
# These packages should already be present, but let's make sure
apt-get install -y \
    bash \
    curl \
    rpm \
    super \
    tar
# MQ Installation
# Download and extract the MQ installation files
mkdir -p /tmp/mq
cd /tmp/mq
curl -LO ${MQ_URL}
tar -zxvf ./*.tar.gz
# Recommended: Create the mqm user ID with a fixed UID and group, so that the
# file permissions work between different images
groupadd --gid 1234 mqm
useradd --uid 1234 --gid mqm --home-dir /var/mqm mqm
usermod -G mqm root
# Configure file limits for the mqm user
echo "mqm       hard  nofile     10240" >> /etc/security/limits.conf
echo "mqm       soft  nofile     10240" >> /etc/security/limits.conf
# Configure kernel parameters to values suitable for running MQ
CONFIG=/etc/sysctl.conf
cp ${CONFIG} /etc/sysctl.conf.bak
sed -i '/^fs.file-max\s*=/{h;s/=.*/=524288/};${x;/^$/{s//fs.file-max=524288/;H};x}' ${CONFIG}
sed -i '/^kernel.shmmni\s*=/{h;s/=.*/=4096/};${x;/^$/{s//kernel.shmmni=4096/;H};x}' ${CONFIG}
sed -i '/^kernel.shmmax\s*=/{h;s/=.*/=268435456/};${x;/^$/{s//kernel.shmmax=268435456/;H};x}' ${CONFIG}
sed -i '/^kernel.shmall\s*=/{h;s/=.*/=2097152/};${x;/^$/{s//kernel.shmall=2097152/;H};x}' ${CONFIG}
sed -i '/^kernel.sem\s*=/{h;s/=.*/=32 4096 32 128/};${x;/^$/{s//kernel.sem=32 4096 32 128/;H};x}' ${CONFIG}
cd /tmp/mq/MQServer
# Accept the MQ license
./mqlicense.sh -text_only -accept
# Install MQ using the RPM packages
rpm -ivh --force-debian ${MQ_PACKAGES}
# Recommended: Set the default MQ installation (makes the MQ commands available on the PATH)
/opt/mqm/bin/setmqinst -p /opt/mqm -i
# Clean up all the downloaded files
rm -rf /tmp/mq
#
# IIB Installation
# Make the destination directory
mkdir -p /opt/ibm
# Download and extract the IIB installation files
mkdir -p /tmp/iib
cd /tmp/iib
curl -LO ${IIB_URL}
tar -zxvf ./*.tar.gz --exclude ${IIB_VERSION}/tools -C /opt/ibm
# Clean up all the downloaded files
rm -rf /tmp/iib
# Clean up unwanted files, to help ensure a smaller image file is created
apt-get clean -y
    The key processing steps are

  • Create a temporary install directory for MQ(/tmp/mq)
  • Download and extract the IBM MQ developer edition
  • Perform MQ setup processing for userids and the setting of kernel parameters
  • Accept the MQ Licence. Note: this contrasts with the IIB case where the licence is accepted later in the processing. See the paragraph below to see why this is the case
  • Clean-up by removing the temporary directory for MQ
  • Create a temporary install directory for IIB(/tmp/iib)
  • Download the IBM Integration Bus developer edition product installation binaries
  • Extract the IBM Integration Bus tar (to /opt/ibm)
  • Clean-up by removing the temporary directory for IIB

The IBM Integration Bus product license is not accepted at this point as accepting the license causes files to be written to /var/mqsi. As there is only a default volume and file system configuration at this point, it would mean /var/mqsi being allocated in the root (/) file system which is not recommended. So by delaying the licence acceptance until the next step, it means that a dedicated file system for /var/mqsi can be allocated prior to accepting the license. This behaviour of what happens at licence acceptance is specific to IBM Integration Bus. The behaviour of IBM MQ at this time is different and it was possible to accept the licence at this time.

Having created the files needed, the command to build the AMI is now invoked from a bash terminal using the command from the folder in which the JSON files are located

packer build iib-mq-aws.json

You should receive output that finishes with something similar to this:

Output of the packer build command

The result of running the packer build command is the creation of a new AMI image (ami-71834f08) that consists of IBM Integration Bus and IBM MQ, installed and partially configured, on Ubuntu 14.04 LTS. Instances of this new AMI can now be run. However, before the image is fully usable and an Integration node defined, some further configuration must be done and this is achieved using a CloudFormation template in AWS.

Step 3: Use AWS CloudFormation to provision one or more instances of the AMI

AWS CloudFormation is a way of allowing the easier configuration of AWS resources. By using this, it is possible to create an instance of an AMI, known as a stack. The template format is specified by Amazon and it is encoded as a JSON or YAML document. In this case the JSON format is used.

The document is used to control the way in which instances of the IBM Integration Bus and IBM MQ AMI created in Step 2 are instantiated. This includes details such as which EBS volumes are to be used, which TCP ports are to be opened for external use and what processing the user supplied configuration scripts will do, including defining Integration nodes and queue managers for example.

The CloudFormation template used in this article is shown below. Remember this is a sample configuration and can be easily changed to meet your own nodes. You could easily define five Integration servers within the Integration node for example, rather than the one used in this case, or define multiple Integration nodes and queue managers.

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "Run an IIB node and MQ Server",
  "Parameters" : {
    "KeyName": {
      "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
      "Type": "AWS::EC2::KeyPair::KeyName",
      "ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
    },
    "AMI": {
      "Description" : "ID of an AMI with IBM MQ pre-installed",
      "Type": "AWS::EC2::Image::Id",
      "ConstraintDescription" : "must be a valid AMI ID."
    },
    "InstanceType" : {
      "Description" : "MQ server EC2 instance type",
      "Type" : "String",
      "Default" : "t2.micro",
      "AllowedValues" : [ "t1.micro", "t2.nano", "t2.micro", "t2.small", "t2.medium", "t2.large", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "m4.large", "m4.xlarge", "m4.2xlarge", "m4.4xlarge", "m4.10xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "g2.8xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"],
      "ConstraintDescription" : "must be a valid EC2 instance type."
    },
    "IIBVersion" : {
      "Description" : "Version of IIB being run in form V.x.x.x",
      "Type": "String",
      "Default" : "10.0.0.10",
      "ConstraintDescription" : "must be a valid Valid version and fixpack level of IBM Integration Bus."
    },
    "IIBNodeName" : {
      "Description" : "Name to use for the IIB node",
      "Type": "String",
      "Default" : "iibNodeX1",
      "ConstraintDescription" : "must be a valid IIB Node name."
    },
    "IIBIntegrationServerName" : {
      "Description" : "Name to use for the IIB Integration Server",
      "Type": "String",
      "Default" : "default",
      "ConstraintDescription" : "must be a valid IIB Integration Server name."
    },
    "QueueManagerName": {
      "Description" : "Name to use for the IBM MQ queue manager",
      "Type": "String",
            "Default" : "iibNodeX1_qmgr",
      "ConstraintDescription" : "must be a valid MQ queue manager name."
    }
  },
  "Resources" : {
     "IIBMQInstance" : {
       "Type" : "AWS::EC2::Instance",
       "Properties" : {
         "InstanceType"   : { "Ref" : "InstanceType" },
         "KeyName"        : { "Ref" : "KeyName" },
         "SecurityGroups" : [ { "Ref" : "IIBMQSecurityGroup" } ],
         "ImageId"        : { "Ref" : "AMI" },
         "UserData": {
           "Fn::Base64": {
             "Fn::Join": [
               "",
               [
                "#cloud-config\n",
                "write_files: \n",
                " - content: |\n",
                "     #!/bin/bash\n",
                "     echo \">>> Starting the configuration of IBM MQ at $(date)\"\n",
                "     while [ $(lsblk|grep xvdf|wc -l) -lt 1 ]; do echo \"Waiting for xdvf to appear\"; sleep 1; done\n",
                "     VOLUME_DEV=\"/dev/xvdf\"\n",
                "     MOUNT_POINT=/var/mqm\n",
                "     mkdir -p ${MOUNT_POINT}\n",
                "     mount ${VOLUME_DEV} ${MOUNT_POINT}\n",
                "     if [[ $? != 0 ]]; then\n",
                "       mkfs.ext4 ${VOLUME_DEV}\n",
                "       echo \"${VOLUME_DEV} ${MOUNT_POINT} ext4 defaults 0 2\" >> /etc/fstab\n",
                "       mount ${MOUNT_POINT}\n",
                "       /opt/mqm/bin/amqicdir -i -f\n",
                "       export MQ_QMGR_NAME=", { "Ref" : "QueueManagerName" }, "\n",
                "       sudo su mqm -c \"crtmqm -q ${MQ_QMGR_NAME}\"\n",
                "     else\n",
                "       export MQ_QMGR_NAME=$(dspmq -n | awk -F '[()]' '{ print $2 }' | head -n 1)\n",
                "       /opt/mqm/bin/amqicdir -i -f\n",
                "     fi\n",
                "     export MQ_QMGR_CONF=/etc/init/strmqm-${MQ_QMGR_NAME}.conf\n",
                "     cp /tmp/mq-upstart-strmqm.conf ${MQ_QMGR_CONF}\n",
                "     sed -i \"s/%QM%/${MQ_QMGR_NAME}/\" ${MQ_QMGR_CONF}\n",
                "     export MQ_QMGR_CONF=/etc/init/endmqm-${MQ_QMGR_NAME}.conf\n",
                "     cp /tmp/mq-upstart-endmqm.conf ${MQ_QMGR_CONF}\n",
                "     sed -i \"s/%QM%/${MQ_QMGR_NAME}/\" ${MQ_QMGR_CONF}\n",
                "     initctl reload-configuration\n",
                "     initctl start strmqm-${MQ_QMGR_NAME}\n",
                "     echo \"<<< Ending the configuration of IBM MQ at $(date)\"\n",
                "   owner: root:root\n",
                "   path: /tmp/configure_mq.sh\n",
                "   permissions: '0700'\n",
                " - content: |\n\n",
                "     description \"Start an IBM MQ queue manager %QM%\"\n\n",
                "     start on runlevel [2345]\n\n",
                "     task\n\n",
                "     console log\n\n",
                "     setuid mqm\n\n",
                "     exec strmqm %QM%\n\n",
                "     setgid mqm\n\n",
                "   owner: root:root\n",
                "   path: /tmp/mq-upstart-strmqm.conf\n",
                "   permissions: '0600'\n",
                " - content: |\n\n",
                "     description \"Stop IBM MQ queue manager %QM%\"\n\n",
                "     start on runlevel [2345]\n\n",
                "     task\n\n",
                "     console log\n\n",
                "     setuid mqm\n\n",
                "     exec endmqm -w %QM%\n\n",
                "     setgid mqm\n\n",
                "   owner: root:root\n",
                "   path: /tmp/mq-upstart-endmqm.conf\n",
                "   permissions: '0600'\n",
                " - content: |\n",
                "     #!/bin/bash\n",
                "     echo \">>> Starting the configuration of IBM Integration Bus at $(date)\"\n",
                "     while [ $(lsblk|grep xvdg|wc -l) -lt 1 ]; do echo \"Waiting for xdvg to appear\"; sleep 1; done\n",
                "     export VOLUME_DEV=/dev/xvdg\n",
                "     export MOUNT_POINT=/var/mqsi\n",
                "     mkdir -p ${MOUNT_POINT}\n",
                "     mount ${VOLUME_DEV} ${MOUNT_POINT}\n",
                "     if [[ $? != 0 ]]; then\n",
                "       mkfs.ext4 ${VOLUME_DEV} 100000\n",
                "       echo \"${VOLUME_DEV} ${MOUNT_POINT} ext4 defaults 0 2\" >> /etc/fstab\n",
                "       mount ${MOUNT_POINT}\n",
                "     fi\n",
                "     export IIB_NODE_NAME=", { "Ref" : "IIBNodeName" }, "\n",
                "     export IIB_INTEGRATION_SERVER_NAME=", { "Ref" : "IIBIntegrationServerName" }, "\n",
                "     export IIB_VERSION=", { "Ref" : "IIBVersion" }, "\n",
                "     export IIB_INSTALL_DIR=/opt/ibm/iib-${IIB_VERSION}\n",
                "     export MQ_QMGR_NAME=", { "Ref" : "QueueManagerName" }, "\n",
                "     # Accept the license - this will also create group a for mqbrkrs and allocate files in /var/mqsi \n",
                "     ${IIB_INSTALL_DIR}/iib make registry global accept license silently\n",
                "     # Recommended: Create the iib user ID with a fixed UID and group, so that the\n",
                "     # file permissions work between different images\n",
                "     useradd --uid 2345 --gid mqbrkrs --home-dir /var/mqsi iib\n",
                "     usermod -G mqbrkrs root\n",
                "     usermod -G mqm iib\n",
                "     # Configure file limits for the iib user\n",
                "     echo \"iib       hard  nofile     10240\" >> /etc/security/limits.conf\n",
                "     echo \"iib       soft  nofile     10240\" >> /etc/security/limits.conf\n",
                "     echo \". ${IIB_INSTALL_DIR}/server/bin/mqsiprofile\" >> ~iib/.bash_profile\n",
                "     chown iib.mqbrkrs ~iib/.bash_profile\n",
                "     sudo su - iib -c \"mqsicreatebroker ${IIB_NODE_NAME} -q ${MQ_QMGR_NAME}\"\n",
                "     sudo su - iib -c \"mqsichangebroker ${IIB_NODE_NAME} -f all\"\n",
                "     sudo su - iib -c \"mqsistart ${IIB_NODE_NAME}\"\n",
                "     sudo su - iib -c \"mqsicreateexecutiongroup ${IIB_NODE_NAME} -e ${IIB_INTEGRATION_SERVER_NAME}\"\n",
                "     echo \"Integration Node is ${IIB_NODE_NAME}\"\n",
                "     export IIB_NODE_CONF=/etc/init/mqsistart-${IIB_NODE_NAME}.conf\n",
                "     cp /tmp/iib-upstart-mqsistart.conf ${IIB_NODE_CONF}\n",
                "     sed -i \"s/%NODE%/${IIB_NODE_NAME}/\" ${IIB_NODE_CONF}\n",
                "     export IIB_NODE_CONF=/etc/init/mqsistop-${IIB_NODE_NAME}.conf\n",
                "     cp /tmp/iib-upstart-mqsistop.conf ${IIB_NODE_CONF}\n",
                "     sed -i \"s/%NODE%/${IIB_NODE_NAME}/\" ${IIB_NODE_CONF}\n",
                "     initctl reload-configuration\n",
                "     initctl start mqsistart-${IIB_NODE_NAME}\n",
                "     echo \"<<< Ending the configuration of IBM Integration Bus at $(date)\"\n",
                "   owner: root:root\n",
                "   path: /tmp/configure_iib.sh\n",
                "   permissions: '0700'\n",
                " - content: |\n\n",
                "     description \"Start IBM Integration Bus node %NODE%\"\n\n",
                "     start on runlevel [2345]\n\n",
                "     task\n\n",
                "     console log\n\n",
                "     exec su - iib bash -c 'mqsistart %NODE%'\n\n",
                "   owner: root:root\n",
                "   path: /tmp/iib-upstart-mqsistart.conf\n",
                "   permissions: '0600'\n",
                " - content: |\n\n",
                "     description \"Stop IBM Integration Bus node %NODE%\"\n\n",
                "     start on runlevel [2345]\n\n",
                "     task\n\n",
                "     console log\n\n",
                "     exec su - iib bash -c 'mqsistop %NODE%'\n\n",
                "   owner: root:root\n",
                "   path: /tmp/iib-upstart-mqsistop.conf\n",
                "   permissions: '0600'\n",
                " - content: |\n\n",
                "     echo \">>> Starting the configuration needed for verification at $(date)\"\n",
                "     echo \"ALTER QMGR CHLAUTH(DISABLED) CONNAUTH(' ')\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     echo \"REFRESH SECURITY TYPE(CONNAUTH)\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     echo \"ALTER CHANNEL(SYSTEM.DEF.SVRCONN) CHLTYPE(SVRCONN) MCAUSER('mqm')\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     echo \"DEFINE LISTENER(LISTENER_PORT_1414) TRPTYPE(TCP) PORT(1414)\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     echo \"START LISTENER(LISTENER_PORT_1414)\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     echo \"DEFINE QLOCAL(IN)\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     echo \"DEFINE QLOCAL(OUT)\" >> /tmp/iib-mq-verify-setup.mqsc\n",
                "     sudo su mqm -c \"runmqsc ${MQ_QMGR_NAME} < /tmp/iib-mq-verify-setup.mqsc\"\n",
                "     echo \"<<< Ending the configuration needed for verification at $(date)\"\n",
                "   owner: root:root\n",
                "   path: /tmp/iib-mq-verify-setup.sh\n",
                "   permissions: '0700'\n",
                "runcmd: \n",
                " - [ /tmp/configure_mq.sh ] \n",
                " - [ /tmp/configure_iib.sh ] \n",
                " - [ /tmp/iib-mq-verify-setup.sh ] \n"
               ]
             ]
           }
         }
       }
     },
     "IIBMQSecurityGroup" : {
       "Type" : "AWS::EC2::SecurityGroup",
       "Properties" : {
       "GroupDescription" : "Enable SSH (22) and IIB (4414, 7080, 7800)",
       "SecurityGroupIngress" : [ {
         "IpProtocol" : "tcp",
         "FromPort" : "22",
         "ToPort" : "22",
         "CidrIp" : "0.0.0.0/0"
       }, {
         "IpProtocol" : "tcp",
         "FromPort" : "1414",
         "ToPort" : "1414",
         "CidrIp" : "0.0.0.0/0"
        }, {
           "IpProtocol" : "tcp",
           "FromPort" : "4414",
           "ToPort" : "4414",
           "CidrIp" : "0.0.0.0/0"
       }, {
         "IpProtocol" : "tcp",
         "FromPort" : "7080",
         "ToPort" : "7080",
         "CidrIp" : "0.0.0.0/0"
       }, {
         "IpProtocol" : "tcp",
         "FromPort" : "7800",
         "ToPort" : "7800",
         "CidrIp" : "0.0.0.0/0"
       } ]
       }
     },
     "MQVolume" : {
       "Type" : "AWS::EC2::Volume",
       "Properties" : {
         "Size" : "1",
         "AvailabilityZone" : { "Fn::GetAtt" : [ "IIBMQInstance", "AvailabilityZone" ]},
         "Tags": [
           { "Key" : "Name", "Value" : "mq" }
         ]
       }
     },
     "MQMountPoint" : {
       "Type" : "AWS::EC2::VolumeAttachment",
       "Properties" : {
         "InstanceId" : { "Ref" : "IIBMQInstance" },
         "VolumeId"  : { "Ref" : "MQVolume" },
         "Device" : "/dev/xvdf"
       }
     },
     "IIBVolume" : {
       "Type" : "AWS::EC2::Volume",
       "Properties" : {
         "Size" : "1",
         "AvailabilityZone" : { "Fn::GetAtt" : [ "IIBMQInstance", "AvailabilityZone" ]},
         "Tags": [
           { "Key" : "Name", "Value" : "iib" }
         ]
       }
     },
     "IIBMountPoint" : {
       "Type" : "AWS::EC2::VolumeAttachment",
       "Properties" : {
         "InstanceId" : { "Ref" : "IIBMQInstance" },
         "VolumeId"  : { "Ref" : "IIBVolume" },
         "Device" : "/dev/xvdg"
       }
     }
  },
  "Outputs" : {
    "InstanceID" : {
      "Description": "The public IP of the MQ server",
      "Value" : { "Fn::GetAtt" : [ "IIBMQInstance", "PublicIp" ]}
    }
  }
}
    The key sections of this template are:

  • Parameters – this describes the parameters that will be passed into the template. This includes the name of the key pair used, the number of the AMI instance, the default names for the Integration node, Integration server, queue manager and the version of IBM Integration Bus being used as this is needed to determine the directory that contains the product binaries. This is not needed for IBM MQ
  • Resources – this describes the resources that will be used in the stack. This includes the name of the public key to be used, the type of EC2 instance, the TCP/IP ports that will be opened, such as the IBM Integration Bus admin port to access the web user interface, the queue manager listener port, EBS volumes to be mounted and some configuration scripts (cloud config) that are specified in one or more files. These configuration scripts are automatically run by AWS as the stack is instantiated. The number and contents of the config files are very specific to the purpose of the AMI and in this case, it is specifically configured to complete the installation of IBM MQ by defining a queue manager (the licence was previously accepted) and for IBM Integration Bus, by accepting the licence and then creating an Integration node. These are fully functioning versions of the products and subject to the licence terms and conditions of the developer editions of these products
  • Outputs - this describes values that can be returned and imported into other stacks or values that are reported on the CloudFormation console or command line. In this example the public IP address of the stack is reported so that the user can log into it using ssh, access the web user interface of IBM Integration Bus, send an HTTP API request to the Integration node and also send and receive MQ messages

The contents of the Parameters and Output sections of the CloudFormation Template are straightforward. The Resources section is more complex and so this will be examined in more detail. For someone who is familiar with using AWS, the processing will be clear but for those new to AWS, it is necessary to understand how configuration processing is completed, so a quick explanation first.

When an instance of an AMI is launched using CloudFormation there is the option to specify user processing, including scripts and configuration files, to perform additional processing – whatever is needed to complete the configuration of the stack – in this case it will be used to complete the configuration of IBM MQ and IBM Integration Bus. The processing to be done is passed in as user data to the CloudFormation template. In this sample it is coded in-line in the template, see UserData, but it could equally have been supplied in a separate file and passed into the CloudFormation command. The scripts are run once on stack instantiation.

For details on the processing that is used in this sample, see the #cloud-config section in the UserData portion that is within the Resources section of the CloudFormation template.

    The write_files directive specifies which files are to be created, the location and access permissions for them. In this template there are seven files:

  1. /tmp/configure_mq.sh – specifies the processing needed to complete the configuration of MQ and create a queue manager
  2. /tmp/configure_iib.sh – specifies the processing needed to complete the configuration of IBM Integration Bus and create an Integration node and Integration server
  3. /tmp/iib-mq-verify-setup.sh – specifies some commands that are used as part of the verification processing in Step 4 to show that the new configuration is operational and able to process messages. This is not part of the product configuration but is needed for some test applications to work
  4. /tmp/mq-upstart-strmqm.conf – specifies the processing to be done by the upstart demon to start the named queue manager after a system start
  5. /tmp/mq-upstart-endmqm.conf – specifies the processing to be done by the upstart demon to stop the named queue manager on system shutdown
  6. /tmp/iib-upstart-mqsistart.conf – specifies the processing to be done by the upstart demon to start the named integration node after a system start
  7. /tmp/iib-upstart-mqsistop.conf - specifies the processing to be done by the upstart demon to stop the named integration node on system shutdown

All of these files are created but not invoked automatically. Script and command invocation is controlled by the runcmd directive. In this case it was decided that three scripts should run:

              "runcmd: \n",
                " - [ /tmp/configure_mq.sh ] \n",
                " - [ /tmp/configure_iib.sh ] \n",
                " - [ /tmp/iib-mq-verify-setup.sh ] \n"

The files used by the upstart demon are not invoked at this time so do not appear in this list. The upstart demon is configured in both of the configure_mq.sh and configure_iib.sh scripts and it is this step that results in those files being used.

By default a CloudFormation stack instance is only allocated a single volume and file system for root (/). Most production systems would use multiple file systems so that space for individual products or instances of web servers/brokers/queue managers etc. can be individually managed without allocating one very large root file system. Using multiple mounted file systems also prevents an Integration node or queue manager filling the root or /tmp file systems with a dump or trace file and so potentially hanging the whole system. So to stop this behaviour the CloudFormation template configures two 1 GB EBS volumes and file systems. The first volume is called /dev/xvdf. A file system is allocated on it on first instantiation of the stack and then mounted as /var/mqm for use by the queue manager. The second volume is called /dev/xvdg. Again a file system is allocated on it and then mounted as /var/mqsi for use by the Integration node. For more detail see MQVolume, MQMountPoint, IIBVolume, IIBMountPoint in the Resources section and the cloud-config scripts /tmp/configure_mq.sh and /tmp/configure_iib.sh. All of these are specified within the Resources section of the template.

    A couple of implementation tips:

  • If you do add other volumes then both the Volume and VolumeAttachment entries must have unique names within the CloudFormation template. If this not the case, the last instance of a name overrides prior use of it
  • During the testing of this configuration it was repeatedly found that the volume attachment to the stack instance was slow and this was leading to configuration being done with only the root file system mounted which was not the intended outcome. In order to overcome this, a loop was added in the configuration scripts to wait until the volume was mounted. The code used in the configuration of MQ was as follows:
    while [ $(lsblk|grep xvdf|wc -l) -lt 1 ]; do echo \"Waiting for xdvf to appear\"; sleep 1; done\n",
    

    There was very similar code used in the script used to configure IBM Integration Bus to check that volume /dev/xvdg is mounted at that time.

Once the template has been created, then instances of the IBM Integration Bus and IBM MQ AMI can be created using a CloudFormation command.

The simplest command that can be used to create an instance is one like this:

aws cloudformation create-stack --stack-name eu-west-1-IIB-MQ-X1  --template-body file://./iib-mq-CF-template.json --parameters ParameterKey=KeyName,ParameterValue=timdunn ParameterKey=AMI,ParameterValue=ami-71834f08
    It specifies:

  • Location of the CloudFormation template file relative to the current directory
  • The key name to be used to communicate with the instances
  • The AMI name. This is the name of the IBM Integration Bus and IBM MQ AMI that was created in Step 2

Other parameters default. This will result in the creation of a stack that has a single Integration node called iibNodeX1, that has a single Integration server within it called default and a queue manager called iibNodeX1_qmgr.

The output from the cloudformation command will be similar to this. It gives the unique identifier of the stack instance.

{
    "StackId": "arn:aws:cloudformation:eu-west-1:299743145002:stack/eu-west-1-IIB-MQ-X1/b894e600-9def-11e7-866b-503ac9eaaafd"
}

An example of a command that overrides the default names for the Integration node, Integration server and queue manager is as follows:

aws cloudformation create-stack --stack-name eu-west-1-IIB-MQ-X2 --template-body file://./iib-mq-CF_template.json --parameters ParameterKey=KeyName,ParameterValue=timdunn ParameterKey=AMI,ParameterValue= ami-63fc311a ParameterKey=IIBNodeName,ParameterValue=iibNodeX2 ParameterKey=IIBIntegrationServerName,ParameterValue=IS1 ParameterKey= QueueManagerName,ParameterValue=iibNodeX2_qm

The details of the stack instance that are created can be viewed using the AWS EC2 Console or using the AWS CloudFormation command line interface using the cloudformation describe-stacks command. For example the output of this command

aws cloudformation describe-stacks --stack-name eu-west-1-IIB-MQ-X1

is as follows:

{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:eu-west-1:299743145002:stack/iibNodeX1/f452af30-9c6c-11e7-9598-503ac9e74c8d",
            "StackName": "iibNodeX1",
            "Description": "Run an IIB node and MQ Server",
            "Parameters": [
                {
                    "ParameterKey": "KeyName",
                    "ParameterValue": "timdunn"
                },
                {
                    "ParameterKey": "QueueManagerName",
                    "ParameterValue": "iibNodeX1_qmgr"
                },
                {
                    "ParameterKey": "IIBVersion",
                    "ParameterValue": "10.0.0.9"
                },
                {
                    "ParameterKey": "IIBNodeName",
                    "ParameterValue": "iibNodeX1"
                },
                {
                    "ParameterKey": "InstanceType",
                    "ParameterValue": "t2.micro"
                },
                {
                    "ParameterKey": "AMI",
                    "ParameterValue": "ami-63fc311a"
                },
                {
                    "ParameterKey": "IIBIntegrationServerName",
                    "ParameterValue": "default"
                }
            ],
            "CreationTime": "2017-09-18T12:29:02.770Z",
            "StackStatus": "CREATE_COMPLETE",
            "DisableRollback": false,
            "NotificationARNs": [],
            "Outputs": [
                {
                    "OutputKey": "InstanceID",
                    "OutputValue": "34.240.182.239",
                    "Description": "The public IP of the MQ server"
                }
            ],
            "Tags": []
        }
    ]
}

At this point there will be a stack instance that has within it a running Integration node and queue manager. By product default it is not possible to access all facilities, this includes the queue manager which has no listener running out of the box. Similarly MQ channel security is fully enabled and so a client cannot access the queue manager without some administrative changes.

For the purposes of illustration and ease of use of this sample, some changes have been made to the configuration and security settings so that it is possible to immediately connect to the Integration node and queue manager without further work.

For an interactive terminal ssh into a shell on the system using port 22. Remember that the AWS Ubuntu AMI’s expose the user id Ubuntu and not root. Use sudo to run commands that require root privilege. You will need your key pass phrase to ssh into the system.

The Integration node web user interface is accessed using port 4414 on the public IP address of the stack instance. To get this address use the aws cloudformation describe-stacks command as was shown above or logon to the EC2 console and look at the CloudFormation display.

To connect to the queue manager use port 1414 which has been opened. In addition a queue manager listener was started and the client channel security settings were disabled using commands in the script /tmp/iib-mq-verify-setup.sh. See the CloudFormation template UserData section or logon on the system to see more details.

Step 4 will walk through a simple verification procedure to show that it is possible to put and get messages from an IBM MQ queue and also invoke a message flow over the HTTP and MQ transports.

Step 4: Verifying the Installation

In this article there will be a simple verification of the installation. It will consist of accessing the Integration node web user interface, writing to and reading from an MQ queue and then running an application using the HTTP and MQ transports.

Part 1. Access the web user interface

The first step is to ensure that the web user interface of the Integration node can be accessed over the public IP address. Use a web browser such as Chrome or Firefox to access http://<public_ip_address:4414>. This should return the web user interface of IBM Integration for the Integration node as show below.

IBM Integration Bus Web User Interface

This has shown the Integration node is indeed running and that it is possible to communicate with it.

Had the Integration node not been running, then you would need to logon on the stack instance and view the contents of /var/log/cloud-init-output.log and/or syslog in order to see why the Integration node was not successfully configured and started.

Part 2. Write a message to an MQ queue and read it back

In this step an MQ client remote to the queue manager will be used to write a message to an MQ queue and then another client will be used to read the message back. This will show that the queue manager can be successfully accessed.

For this test a remote Windows 7 system will be used with the IBM MQ v9.0.0.3 CD client installed.

To write and read MQ messages, two MQ sample programs are used. These are amqsputc to put messages to a named queue and amqsetc to read messages from a named queue.

The first step needed on the client machine is to open a command window and check that the MQ bin and MQ samples bin directory are in the PATH environment variable. If not, use this command to add them (assuming a default installation).

SET PATH=%PATH%;C:\Program Files\IBM\MQ\bin;C:\Program Files\IBM\MQ\tools\c\samples\bin

Then set the MQSERVER environment variable to the public IP address of the stack instance and port number of the queue manager, so in this instance it was set as this;

SET MQSERVER=SYSTEM.DEF.SVRCONN/TCP/54.72.55.160(1414)

Then invoke the amqsputc command to write some messages to the OUT queue on the queue manager. Follow this by running amqsgetc to read the messages back. This picture shows that being done.

Verifying queue write and read activity

At this point it has been possible to write messages to a queue on iibNodeX1_qmgr and read them back. The next step is to deploy an application to the Integration node.

Part 3. Deploy the Verification Application

This section will use a simple IBM Integration Bus Application called IN_OUT_HTTP_MQ that will send a response message to show what time the message flow was invoked. This can be invoked over the HTTP and MQ transports.

Navigate to the Deploy operation in the web user interface for the Integration node.

Selecting the Deploy Operation

Then locate the bar file on the local machine and deploy it. After you receive message to say that the bar file deployed successfully, verify that it can be seen in the web user interface. There should be a display similar to this:

Verifying Successful Deploy Operation

At this point the application can now be invoked over the HTTP and MQ transports.

Part 4. Invoking the Application over HTTP

When using the HTTP transport, the application IN_OUT_HTTP_MQ is invoked using the URI http://<public_ip_address>:7080/INOUT

The URI is typed in a web browser such as Chrome or Firefox and a response received.

The output is similar to this:

Output from the HTTP Request

If the application does not successfully return a response you should analyse the problem using the normal IBM Integration Bus problem determination facilities.

Part 5. Invoking the Application over MQ

When using the MQ transport, the application IN_OUT_HTTP_MQ is invoked by writing a message to the queue IN and reading the response from OUT.

Using the amqsputc and amqsgetc samples, applications messages are written to the queue IN and read from the queue OUT. This picture shows this being done:

Output from the IN_OUT_HTTP_MQ message flow using MQ transport

If the application does not successfully return a response you should analyse the output using the regular IBM Integration Bus and IBM MQ problem determination facilities.

At this point an application has been successfully invoked in a deployed instance of IBM Integration Bus running with AWS using both the HTTP and MQ transports and this simple verification procedure is complete.

This example used the eu-west-1 region but any other that supports the required level of base AMI could also have been used. To do this, you would also need to repeat the packer build stage to produce an AMI that was located in the new region and to then use that new AMI identifier when creating a new stack instance.

Limitations of the AMI

The procedure documented in this article is intended to show you how to build and deploy an AMI instance in AWS. It is provided for guidance and education purposes only. There are a number of limitations with the procedure (which are documented in the list below) but it does show you how to script the building and deployment of an AMI image on which IBM Integration Bus and IBM MQ are installed. Making the necessary modifications that would give you a production ready image for your own environment is a delta on this basic deployment and hopefully you can see how to extend the process.

    Known limitations:

  • Uses the developer download version of IBM Integration Bus and IBM MQ which are taken from an IBM site as the installation image to use. These versions are subject to the developer edition licence terms. These images could easily be replaced by a locally held full installation image by editing the file iib-mq-aws.json and changing the value of IIB_URL
  • Limited security is in place using the AWS key pair. No security profiles have been created for users of IBM Integration Bus for example
  • Dedicated file systems /var/mqsi and /var/mqm are allocated and used but in a typical production system others are used
  • The storage used is EBS. For a more scalable and robust deployment it is likely that you would look at using EFS
  • The AMI is built as a single instance in one region. It is only deployed to a single AWS region at a time and does not have availability groups defined for example
  • The AMI with IBM Integration Bus and IBM MQ installed is basic. It is an Ubuntu image with only IBM Integration Bus and IBM installed. In a typical production environment, there would be other applications and utilities that also need to be installed beside the broker. This is easy to add though and you would do it by extending the iib-mq-install.sh script and by extending the cloud config scripts that are run when the CloudFormation stack is initiated.

Summary

This article has shown how it is possible to build an AMI with IBM Integration Bus v10.0.0.10 and IBM MQ v9.0.0.3 CD installed and how to then create instances of that AMI using an AWS CloudFormation template and commands.

Building the AMI with IBM Integration Bus and IBM MQ only needs to be done once. Multiple instances of the AMI can then be created as required.

Resources

The templates used in this article are available for download in the iib-aws repository in the Open Technologies for Integration(ot4i) github server. See the README.md file at this link for more details.

Join The Discussion

Your email address will not be published. Required fields are marked *