In this post I will show you how you can push a native C client for MQ (in this case I’ll be using the amqsgetc sample) as a Bluemix Cloud Foundry app, and connect it to an on-premise queue manager via the Secure Gateway. We’ll be using a variety of tools as we go through the necessary steps so it may be useful to familiarise yourself with some of the following tools:
- The Bluemix CLI
- Cloud Foundry apps & manifests
- The MQ redistributable client
- The Bluemix Secure Gateway
I’ll take you through the steps in detail but for those who don’t want to go through all of the detail, here’s a quick summary.
- Install the Bluemix CLI and Cloud Foundry plugin. Login to Bluemix with “bx cf login” or “bx cf login –sso”
- Create a Cloud Foundry app, specifying the binary buildpack and including the MQ redistributable client files
- Create a Secure Gateway instance in Bluemix. Install one of the SG clients (in my case, Docker), and start it
- Define a Secure Gateway destination to give the CF app a route from Bluemix to your on-premise network
- Add the Secure Gateway destination to the CF app manifest so amqsgetc knows the host and port of the SG in Bluemix
- Specify a SG access control list (ACL) in the SG client, giving access to the host and port your queue manager is running on
- Push the CF app to Bluemix
- Put some messages to SYSTEM.DEFAULT.LOCAL.QUEUE and check that they appear in the amqsget Cloud Foundry logs.
This is a fairly simple example that uses the MQ sample amqsgetc to demonstrate how you can offload some of your on-premise workload to the cloud. All amqsgetc does is to read messages from a queue and write them to standard out. You could replace amqsgetc with any business application that uses an MQ client.
If you want more detailed instructions, keep reading.
Install the Bluemix CLI
This step gives you the bx command through which you can administer Bluemix. There are various plugins to the bx command. They add options that let you:
- administer Bluemix containers
- configure Bluemix VPNs
- manage Bluemix private networks
We need to use the cf plugin to administer Cloud Foundry applications in Bluemix. If you install the Bluemix CLI from here it comes with the Cloud Foundry CLI pre-bundled. Download and install the Bluemix CLI.
Once it is installed, login to Bluemix. If you don’t already have a Bluemix account go to bluemix.net to sign up for a free trial period.
If you have a non-federated ID (most users) you can login by running “bx login” (to log in to the Bluemix US-South region) or “bx login -a <region-api-endpoint>” to connect to a specific Bluemix region.
Note: If you have a federated Bluemix account, you will need to specify the “-sso” and follow the instructions to perform a single-sign-on.
Create a Cloud Foundry app
Create an empty directory called mq-cf-app and inside it create a manifest.yml file, and an empty directory called mqlibs.
In a later step we will put the MQ client libraries into the mqlibs folder.
The manifest.yml file should contain the following:
– name: amqsget
¬†¬†command: while true; do mqlibs/samp/bin/amqsgetc SYSTEM.DEFAULT.LOCAL.QUEUE; sleep5; done
The valid Cloud Foundry attributes are documented here. The ones we’ve used are:
This must be present, and begins the list of applications that this manifest will deploy. It is possible to have multiple applications within a single manifest, but in this case we’re only going to deploy one.
The name of the application. This isn’t the same as the name of the command we’re going to run, it is simply a way of distinguishing this application from others you might have deployed.
By default Cloud Foundry assumes that the application you’re deploying will need to accept network connections from other applications. This would be the case if you were deploying a web server. However in our case, we’re running amqsgetc as a standalone application that doesn’t accept network connections, so we set this attribute to false. This means Cloud Foundry won’t generate a URL or network route for external applications to connect to our app.
Cloud Foundry will attempt to detect which type of buildpack to provision to run an app in. Bluemix has some built-in build packs. If we deployed a .war file it would auto-detect that we had pushed a web app, and automatically provision the built-in Liberty buildpack to run it in. If a built-in buildpack isn’t available for a particular application you can tell Cloud Foundry to use some other third-party buildpack.¬†For native C applications there is no built-in buildpack so we are going to tell Bluemix to use the “binary” buildpack provided by Cloud Foundry. The buildpack attribute in the manifest specifies where Bluemix can download the buildpack from.
This is the command that will be executed when the application is launched
This tells Cloud Foundry how to tell if the application is healthy and, therefore, doesn’t need restarting. If we don’t specify health-response-type as process Cloud Foundry defaults to checking that any port we open is still being used. Since amqsgetc doesn’t accept connections or data over a port Cloud Foundry will assume amqsgetc needs restarting as soon as it starts.
This is where we set environment variables that need to be set when amqsgetc¬†is run. There is an entry to configure the LD_LIBRARY_PATH so amqsgetc can locate the necessary MQ libraries, and an entry for MQSERVER to specify the MQ channel and host to connect to.
We’ll look at the value of MQSERVER more closely later in the article.
Create and configure a Secure Gateway service
The Bluemix Secure Gateway provides a secure tunnel between applications running in Bluemix and other on-premise or cloud environments. You create an instance of the Secure Gateway in Bluemix, and then run a Secure Gateway client in the location you want to create a secure tunnel to. In this case I’m running the Secure Gateway client on a Linux virtual machine running on my laptop. My queue manager is running on the same Linux VM, so once the tunnel has been set up the Secure Gateway client will simply connect to localhost port 1414 on behalf of amqsgetc and tunnel data to it.
Note: because I use the Docker Secure Gateway client I’ll actually configure it to connect to 172.17.0.1, the address of my VM on the docker network, but effectively it will connect to localhost on the VM.
Start by creating a Secure Gateway service in your Bluemix organisation.
You can configure the Secure Gateway service with multiple gateway definitions to create connections to different environments, but if you are using the free plan you are limited to a single gateway. That’s enough to try out this scenario so create a new gateway and give it a name. I’ve called mine “amqsget-gateway”. Leave the default options ticked.
Once you have created a gateway you need to connect a client. Pressing “Connect Client” presents you with options for installing and running a client. Remember that the client will need to be installed and run on an on-premise environment that has network access to the queue manager you want to connect to. I’m running the Docker client but you can choose any client that suits the platform you’re running on. The Gateway ID and Security Token that have been generated are displayed – you will need to use these when you start the Secure Gateway client.
Once you have started a client and it has connected successfully, you should see the client in the Secure Gateway dashboard.
Now we can create a destination in the Secure Gateway dashboard that specifies the on-premise host and port we would like to connect to. When amqsgetc connects to the Secure Gateway service, it will be tunneled through the Secure Gateway to the host and port that we configure in this step. The destination wizard will walk you through the steps needed to define a route. For this article I selected “On-Premises” as the location of my resource (the queue manager), 172.17.0.1 as the host, 1414 as the port.
Note: be wary of specifying “localhost” as the hostname. If you are using the Secure Gateway Docker client for example, localhost will resolve to the docker container itself, not the host you have launched it from.
When you have created a destination you can view the cloud host and port that amqsgetc must connect to. This is a host and port that is accessible to your Cloud Foundry application:
Configuring amqsgetc to connect to the Secure Gateway
When amqsgetc connects to the Secure Gateway cloud host and port (in my case¬†cap-sg-prd-1.integration.ibmcloud.com:17036) the connection is tunneled through to the gateway client running on-premise, and routed to the endpoint you configured in the destination (in my case 172.17.0.1 port 1414) resulting in a connection to the on-premise queue manager.
There are several ways to configure where an MQ client should connect to, but one simple way is to set the MQSERVER environment variable. The variable should be set to:
<mq-channel-name>/<network protocol>/<host & port>
For our Cloud Foundry application we need amqsgetc to connect to the Secure Gateway cloud host so in the manifest.yml set the MQSERVER environment variable to:
(obviously you must specify a different SVRCONN channel if you’re not using SYSTEM.DEF.SVRCONN)
Configure the Secure Gateway access control list (ACL)
By default, Secure Gateway clients prevent access to any host or port on your on-prem network, even though you have defined a destination in the Secure Gateway dashboard. To allow the Secure Gateway client to connect to your local queue manager you need to specify an ACL that allows it to connect to the IP address your queue manager is running on. In my case this is 172.17.0.1.
The different Secure Gateway clients are configured with ACLs in different ways. If you have used the Docker client it launches a command-line interface as soon as you start it. Entering “acl allow 172.17.0.1:1414” into the Docker CLI allows access to that host and port. If you are using the Linux client you will need to specify an ACL configuration file when installing the client. More information on how to configure ACLs can be found in the Secure Gateway docs.
Create the application files and directory structure
The application we are going to run in Cloud Foundry is the MQ sample amqsgetc. However we must¬†push more than just the amqsgetc binary to Cloud Foundry. We need to include the MQ libraries it requires to run.
When MQ 8 FP 4 was released, a new redistributable MQ client bundle was made available to make it easier to run MQ client applications and copy the necessary MQ libraries around without having to run a full MQ install. You can read more about the redistributable clients in the MQ KnowledgeCenter. There are several versions of the package available, including a redistributable version of the MQ Managed File Transfer agent and a redistributable MQ Java client package. For this demo we need the MQ C package for Linux x64. Download the package and unzip it to a new directory.
The package contains all of the files and libraries necessary to run a variety of native MQ client applications. We can dramatically reduce its size by going into the bin directory and running the command genmqpkg.sh. Answer no to all of the components it asks about, with the exception of the MQ samples which we want to be included. The command asks you for a target directory to put the reduced package into. Specify the mqlibs directory you created alongside your manifest.yml. When the tool completes, the directory structure for your Cloud Foundry application should now look like this:
Specifying the Cloud Foundry command to run
For a client application that was designed to be long-lived we could just specify the name of the executable as the command to run. However the amqsgetc sample terminates after no new messages have been received after 30 seconds. Because Cloud Foundry automatically restarts stopped applications, this would cause amqsgetc to run for 30 seconds, then to terminate, then be restarted by Cloud Foundry.
To stop Cloud Foundry having to constantly restart the application we will run it within a command that automatically re-runs amqsgetc a few seconds after it finishes. If you wanted to remove this from the example you could recompile amqsgetc without the 30 second timeout.
For this article specify the following as the value of the command in the manifest:
while true; do mqlibs/samp/bin/amqsgetc SYSTEM.DEFAULT.LOCAL.QUEUE; sleep 5 ;done
Pushing the application to Cloud Foundry
With the manifest.yml configured correctly, run the following command from the directory where manifest.yml is:
bx cf push
This will upload all of the files in the directory, and launch the command specified in manifest.yml. You should see output like this if the application was pushed successfully:
When the application has started amqsgetc will print to standard out any messages it receives. Try putting some messages to the SYSTEM.DEFAULT.LOCAL.QUEUE and then run the following command to view the logs of the Cloud Foundry app:
bx cf logs amqsget
Now you’ve run an MQ sample in Bluemix Cloud Foundry, try modifying your setup to:
- Reduce the amount of memory the application requires by setting the memory attribute in the manifest. The default of 1GB is far more than amqsgetc¬†needs.
- Use your own MQ client application instead of amqsgetc
- Run more than 1 instance of the app by setting the instance attribute in the manifest
- Also try running the “bx cf scale” command to manually request new instances of the application