A Composed Sample Application with DataPower and MQ

This sample demonstrates how to use multiple containers, all orchestrated by Docker Compose, to create an application that includes DataPower, MQ, and Apache with integrated testing using SoapUI and curl. This sample includes everything you need – the load driver, the DataPower, and the MQ back-end. Everything is there – config, stylesheets, docker-compose.yml and Dockerfiles. This is a highly customizable example and it is encouraged that you play around with it. Read on and I’ll show you how the application is composed!

Prerequisites

Before you attempt this tutorial, please be sure that you:

  • Read, and better yet attempt, the Docker Compose Hello World Application sample. This is a great starting point that offers some of the detailed framework of how DataPower containers can be created and used in a continuous integration system.
  • For your convenience, I created and used a Docker MQ base image. If you wish to run this step manually, please follow the instructions on ibmcom/mq on Docker Hub to create your own base Docker MQ image.

The Composed Application

No rock band here, just a simple message. Harley’s Hello World example (mentioned in the Prerequisite section) goes into great detail about the various elements of his project. In order to avoid redundancy, I will not go into such detail here, but I will still cover the whole project.

First, let me set the stage. I have an existing DataPower deployment and I want to begin to adopt continuous integration and continuous delivery. I plan to deploy containers in my development and test environments, while final regression and deployment environments are physical appliances. In production, I store stylesheets and schema, used by DataPower processing policies, on a HTTP server. Since I want to be able to export from my container and import into production, it is most straightforward to also load these files from a HTTP server in develpment and test. Fortunately, with Compose this is easy!

Let’s take a look at the docker-compose.yml file that orchestrates the entire application.

version: '2'
services:
    mq:
      build: mq
    http:
      build: http
    datapower:
      build: datapower
      depends_on:
       - mq
       - http
    soapui:
      image: ddavison/soapui:latest
      depends_on:
       - datapower
    curldriver:
      build: curldriver
      depends_on:
       - soapui

In all, the application consists of five services, which build five images, and run five containers.

  • The curldriver service connects to soapui which drives the transaction. All curldriver does is send simple curl requests.
  • The soapui service uses an off-the-shelf SoapUI image from Docker Hub and adds configuration required to test the DataPower MQ service. The SoapUI request is triggered by curldriver, and it connects to DataPower.
  • The datapower service uses a DataPower Gateway from ibmcom/datapower. Configurations are added, including an application domain that contains a multi-protocol gateway. The service processes requests from soapui, using stylesheets from http, and sends the request to mq
  • The http service uses an apache server image from Docker Hub. Stylesheets and schema are placed on the server, which are called by datapower.
  • The mq service uses a base MQ queue manager from Docker Hub. Configuration is added and the service accepts messages from datapower.

The general flow is an HTTP request is sent via curl to a SoapUI container, which sends a test request to DataPower. The request is transformed by a processing policy in DataPower, which uses style sheets and schema from a web server. Finally, the resulting message is sent from DataPower to a back end MQ queue manager. MQ sends a response back to Datapower who fowards the message back to the requester, SoapUI. An assertion on the response is evaluated by SoapUI and a message is returned to the composed application’s log with the outcome of the test.

From the docker-compose.yml file, you can see that the services have dependencies on one another. As in the Hello World Docker Compose example, networking for the application is completely controlled by Docker Compose. Compose gives each copy of the application its own network. All of the containers are aware of each other in the network and the different nodes can be referred by one another as they are named by the docker-compose.yml.

Run the Application

In order to run the application, you need to get the source. Run the following commands (note: if you have completed the docker-compose hello world sample, you need only run the second command from the datapower-tutorials directory):

git clone https://github.com/ibm-datapower/datapower-tutorials.git
cd datapower-tutorials/datapower-docker-compose-dp-mq/src

The src directory contains the docker-compose.yml file that will build the composed application. We’ll build the application with the following command:

docker-compose build

This builds four images as you can see below:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
src_curldriver      latest              cb496bbd7b7c        12 minutes ago      184.2 MB
src_datapower       latest              dd56a9be74e0        14 minutes ago      716.7 MB
src_mq              latest              b003e8e70670        16 minutes ago      844.4 MB
src_http            latest              e55379457250        16 minutes ago      244.6 MB

Note that src_ is appended to each image name. This comes from the src context directory name. Also, the src_soapui image is not created at docker-compose build-time, as it is an image from Docker Hub, and will be pulled during the docker-compose up command.

Next, we will run our composed application, which will then process the request.

docker-compose up

When the docker-compose up command is issued, you can follow along as the project containers are created and sample messages are sent through the work flow:

autobuild:src$ docker-compose up
Pulling soapui (ddavison/soapui:latest)...
latest: Pulling from ddavison/soapui
357ea8c3d80b: Pull complete
52befadefd24: Pull complete
3c0732d5313c: Pull complete
557cb7f84eb9: Pull complete
7bbd9fac5727: Pull complete
15f5ec8580f1: Pull complete
b262ae65d728: Pull complete
d2cd040ac8b2: Pull complete
ebb4e3f1e29f: Pull complete
75d31cfdd3e1: Pull complete
Digest: sha256:7f11236a781c8cdf74092593afda1299ad7372e15c72ca8add22df892af1cc6b
Status: Downloaded newer image for ddavison/soapui:latest
Creating src_http_1
Creating src_mq_1
Creating src_datapower_1
Creating src_soapui_1
Creating src_curldriver_1
Attaching to src_http_1, src_mq_1, src_datapower_1, src_soapui_1, src_curldriver_1
.....
.....
soapui_1      | 172.19.0.6 - - [07/Jan/2017 22:42:39] "POST / HTTP/1.1" 200 -
curldriver_1  | 22:42:38,819 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 200 OK
curldriver_1  | SUCCESS! Exiting
src_curldriver_1 exited with code 0

src_soapui_1 has assertions set up to validate the message being returned. When the assertions are met, a 200 - OK message is returned to curldriver_1 as can be seen in the output above.

Depending on the system on which you are deploying, the first message or two may fail to be processed as the application may still be building. This is easily detectable in the console logs.

After the first successful response is received the curldriver script exits. You may end the test with the docker command ctrl+c, which will exit the application and stop the containers. To rerun the test, run the docker-compose up command again.

The Request Flow

Here I’ll outline the travels of the request message created in the docker-compose application.

First from the Dockerfile, the src_curldiriver image is configured to run the following shell script which sends a curl command via HTTP to src_soapui.

#!/bin/bash
# set env vars to modify operation
# TIMEOUT=<time in seconds> for amount of time to continue to retry
# CONTINUOUS=true to continuously test, otherwise exits immediately on success

if [ "$TIMEOUT" -a "$TIMEOUT" -gt 0 ]
then
  STOPTIME=$(( $(date +%s) + $TIMEOUT ))
else
  STOPTIME=9999999999
fi

RC=1

while [ $(date +%s) -lt $STOPTIME ]
do
  if curl --silent --insecure --form "project=@/usr/local/bin/grow-soapui-project.xml" --form "suite=sendRequestGrow" http://soapui:3000 | grep "200 OK"
  then
    RC=0
    if [ "$CONTINUOUS" != "true" ]
    then
      echo "SUCCESS! Exiting"
      exit $RC
    fi
  else
    echo curl error $?
  fi
  sleep 5
done

SoapUI is acting as a proxy. It takes a http request form curl that contains the soapui project and uses the project to construct the request or requests that it then sends to DataPower. We don’t need to change anything in SoapUI! The project file is passed in via curl, so the off-the-shelf soapui DockerHub image is all we need.

The script is set to send a request message every five seconds until a successful transaction is completed. Note that the console log may show a few request failures as the application comes up.

src_datapower_1 receives messages from src_soapui_1 in the growTest application domain. The processing policy retrieves various stylesheets and schema from src_http_1 and transforms the message. The policy itself is simple: As soapui’s messages pass through each rule in the policy, the number of nodes and elements increase. At the end, the transformed message reports its size. DataPower then routes the transformed message to the src_mq_1 back end.

After the transformed message is placed on a queue in src_mq_1 that could be the end of the story. However, to avoid the queue getting full on a long running test, I set up a dummy service on src_datapower_1 that clears the queue and throws the message away.

##Cleaning Up When you have completed your testing, you can remove the containers created during the exercise with the following command:

docker-compose rm

You can rerun the application by issuing the docker-compose up command again.

Making Changes- You have the power

The power to make changes in this sample is in your hands. The construction of the application lends itself to customization. Let us say you want to use Docker containers for your distributed application’s entire life cycle (dev, test, prod). Contrary to my example, In a truly dockerized application, it would be more common to store the stylesheet and schema files natively on DataPower in the local: directory eliminating the HTTP container altogether. Additionally, the processing policy in the DataPower configuration can be replaced with your own; the SoapUI project can be changed to run a different test or additional requests could be added; the curl driver parameters can be adjusted to change the rate and duration of test messages; multiple MQ queue managers can be added. The list is endless and completely up to you.

Conclusion

I was able to use off-the-shelf Docker images and tools to create a portable containerized application in a very short period of time. I can run it wherever I can run Docker Compose. All the configuration I require is in version control. I can generate relatively complicated requests using SoapUI and I can use DataPower, MQ, and potentially other IBM Middleware as first-class citizens in my containerized applications. And if I can do it, you can too!

I hope you find this sample helpful. I appreciate your time and attention and look forward to your feedback!

1 comment on"A Docker Composed Sample Application with DataPower and MQ"

  1. Tom Watson March 30, 2017

    awesome lab. I did have one issue.
    The creation of the MQ container was failing with “AMQ6024 Insufficient resources to complete request…”. My virtual machine that is hosting docker did not have recommended BM WebSphere MQ System V IPC resources, in particular shared memory and semaphores.

    …The minimum configuration for IBM WebSphere MQ for these resources is as follows:

    kernel.shmmni = 4096
    kernel.shmall = 2097152
    kernel.shmmax = 268435456
    kernel.sem = 500 256000 250 1024
    fs.file-max = 524288
    kernel.pid-max = 120000
    kernel.threads-max = 48000
    per https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ins.doc/q008550_.htm

    After changing those kernel settings, I was able to complete the rest of your lab.

Join The Discussion

Your email address will not be published. Required fields are marked *