A Composed Sample Application with DataPower and MQ
This sample demonstrates how to use multiple containers, all orchestrated by Docker Compose, to create an application that includes DataPower, MQ, and Apache with integrated testing using SoapUI and curl. This sample includes everything you need â€“ the load driver, the DataPower, and the MQ back-end. Everything is there â€“ config, stylesheets,
Dockerfiles. This is a highly customizable example and it is encouraged that you play around with it. Read on and Iâ€™ll show you how the application is composed!
Before you attempt this tutorial, please be sure that you:
- Read, and better yet attempt, the Docker Compose Hello World Application sample. This is a great starting point that offers some of the detailed framework of how DataPower containers can be created and used in a continuous integration system.
- For your convenience, I created and used a Docker MQ base image. If you wish to run this step manually, please follow the instructions on ibmcom/mq on Docker Hub to create your own base Docker MQ image.
The Composed Application
No rock band here, just a simple message. Harleyâ€™s Hello World example (mentioned in the Prerequisite section) goes into great detail about the various elements of his project. In order to avoid redundancy, I will not go into such detail here, but I will still cover the whole project.
First, let me set the stage. I have an existing DataPower deployment and I want to begin to adopt continuous integration and continuous delivery. I plan to deploy containers in my development and test environments, while final regression and deployment environments are physical appliances. In production, I store stylesheets and schema, used by DataPower processing policies, on a HTTP server. Since I want to be able to export from my container and import into production, it is most straightforward to also load these files from a HTTP server in develpment and test. Fortunately, with Compose this is easy!
Letâ€™s take a look at the
docker-compose.yml file that orchestrates the entire application.
version: '2' services: mq: build: mq http: build: http datapower: build: datapower depends_on: - mq - http soapui: image: ddavison/soapui:latest depends_on: - datapower curldriver: build: curldriver depends_on: - soapui
In all, the application consists of five services, which build five images, and run five containers.
curldriverservice connects to
soapuiwhich drives the transaction. All
curldriverdoes is send simple curl requests.
soapuiservice uses an off-the-shelf SoapUI image from Docker Hub and adds configuration required to test the DataPower MQ service. The SoapUI request is triggered by
curldriver, and it connects to DataPower.
datapowerservice uses a DataPower Gateway from ibmcom/datapower. Configurations are added, including an application domain that contains a multi-protocol gateway. The service processes requests from
soapui, using stylesheets from
http, and sends the request to
httpservice uses an apache server image from Docker Hub. Stylesheets and schema are placed on the server, which are called by
mqservice uses a base MQ queue manager from Docker Hub. Configuration is added and the service accepts messages from
The general flow is an HTTP request is sent via curl to a SoapUI container, which sends a test request to DataPower. The request is transformed by a processing policy in DataPower, which uses style sheets and schema from a web server. Finally, the resulting message is sent from DataPower to a back end MQ queue manager. MQ sends a response back to Datapower who fowards the message back to the requester, SoapUI. An assertion on the response is evaluated by SoapUI and a message is returned to the composed applicationâ€™s log with the outcome of the test.
docker-compose.yml file, you can see that the services have dependencies on one another. As in the Hello World Docker Compose example, networking for the application is completely controlled by Docker Compose. Compose gives each copy of the application its own network. All of the containers are aware of each other in the network and the different nodes can be referred by one another as they are named by the
Run the Application
In order to run the application, you need to get the source. Run the following commands (note: if you have completed the docker-compose hello world sample, you need only run the second command from the datapower-tutorials directory):
git clone https://github.com/ibm-datapower/datapower-tutorials.git cd datapower-tutorials/datapower-docker-compose-dp-mq/src
The src directory contains the
docker-compose.yml file that will build the composed application. Weâ€™ll build the application with the following command:
This builds four images as you can see below:
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE src_curldriver latest cb496bbd7b7c 12 minutes ago 184.2 MB src_datapower latest dd56a9be74e0 14 minutes ago 716.7 MB src_mq latest b003e8e70670 16 minutes ago 844.4 MB src_http latest e55379457250 16 minutes ago 244.6 MB
Note that src_ is appended to each image name. This comes from the src context directory name. Also, the
src_soapui image is not created at docker-compose build-time, as it is an image from Docker Hub, and will be pulled during the docker-compose up command.
Next, we will run our composed application, which will then process the request.
When the docker-compose up command is issued, you can follow along as the project containers are created and sample messages are sent through the work flow:
autobuild:src$ docker-compose up Pulling soapui (ddavison/soapui:latest)... latest: Pulling from ddavison/soapui 357ea8c3d80b: Pull complete 52befadefd24: Pull complete 3c0732d5313c: Pull complete 557cb7f84eb9: Pull complete 7bbd9fac5727: Pull complete 15f5ec8580f1: Pull complete b262ae65d728: Pull complete d2cd040ac8b2: Pull complete ebb4e3f1e29f: Pull complete 75d31cfdd3e1: Pull complete Digest: sha256:7f11236a781c8cdf74092593afda1299ad7372e15c72ca8add22df892af1cc6b Status: Downloaded newer image for ddavison/soapui:latest Creating src_http_1 Creating src_mq_1 Creating src_datapower_1 Creating src_soapui_1 Creating src_curldriver_1 Attaching to src_http_1, src_mq_1, src_datapower_1, src_soapui_1, src_curldriver_1 ..... ..... soapui_1 | 172.19.0.6 - - [07/Jan/2017 22:42:39] "POST / HTTP/1.1" 200 - curldriver_1 | 22:42:38,819 DEBUG [SoapUIMultiThreadedHttpConnectionManager$SoapUIDefaultClientConnection] Receiving response: HTTP/1.1 200 OK curldriver_1 | SUCCESS! Exiting src_curldriver_1 exited with code 0
src_soapui_1 has assertions set up to validate the message being returned. When the assertions are met, a
200 - OK message is returned to
curldriver_1 as can be seen in the output above.
Depending on the system on which you are deploying, the first message or two may fail to be processed as the application may still be building. This is easily detectable in the console logs.
After the first successful response is received the curldriver script exits. You may end the test with the docker command ctrl+c, which will exit the application and stop the containers. To rerun the test, run the
docker-compose up command again.
The Request Flow
Here Iâ€™ll outline the travels of the request message created in the docker-compose application.
First from the Dockerfile, the
src_curldiriver image is configured to run the following shell script which sends a curl command via HTTP to
#!/bin/bash # set env vars to modify operation # TIMEOUT=<time in seconds> for amount of time to continue to retry # CONTINUOUS=true to continuously test, otherwise exits immediately on success if [ "$TIMEOUT" -a "$TIMEOUT" -gt 0 ] then STOPTIME=$(( $(date +%s) + $TIMEOUT )) else STOPTIME=9999999999 fi RC=1 while [ $(date +%s) -lt $STOPTIME ] do if curl --silent --insecure --form "project=@/usr/local/bin/grow-soapui-project.xml" --form "suite=sendRequestGrow" http://soapui:3000 | grep "200 OK" then RC=0 if [ "$CONTINUOUS" != "true" ] then echo "SUCCESS! Exiting" exit $RC fi else echo curl error $? fi sleep 5 done
SoapUI is acting as a proxy. It takes a http request form curl that contains the soapui project and uses the project to construct the request or requests that it then sends to DataPower. We donâ€™t need to change anything in SoapUI! The project file is passed in via curl, so the off-the-shelf soapui DockerHub image is all we need.
The script is set to send a request message every five seconds until a successful transaction is completed. Note that the console log may show a few request failures as the application comes up.
src_datapower_1 receives messages from
src_soapui_1 in the
growTest application domain. The processing policy retrieves various stylesheets and schema from
src_http_1 and transforms the message. The policy itself is simple: As soapuiâ€™s messages pass through each rule in the policy, the number of nodes and elements increase. At the end, the transformed message reports its size. DataPower then routes the transformed message to the
src_mq_1 back end.
After the transformed message is placed on a queue in s
rc_mq_1 that could be the end of the story. However, to avoid the queue getting full on a long running test, I set up a dummy service on
src_datapower_1 that clears the queue and throws the message away.
##Cleaning Up When you have completed your testing, you can remove the containers created during the exercise with the following command:
You can rerun the application by issuing the docker-compose up command again.
Making Changes- You have the power
The power to make changes in this sample is in your hands. The construction of the application lends itself to customization. Let us say you want to use Docker containers for your distributed applicationâ€™s entire life cycle (dev, test, prod). Contrary to my example, In a truly dockerized application, it would be more common to store the stylesheet and schema files natively on DataPower in the local: directory eliminating the HTTP container altogether. Additionally, the processing policy in the DataPower configuration can be replaced with your own; the SoapUI project can be changed to run a different test or additional requests could be added; the curl driver parameters can be adjusted to change the rate and duration of test messages; multiple MQ queue managers can be added. The list is endless and completely up to you.
I was able to use off-the-shelf Docker images and tools to create a portable containerized application in a very short period of time. I can run it wherever I can run Docker Compose. All the configuration I require is in version control. I can generate relatively complicated requests using SoapUI and I can use DataPower, MQ, and potentially other IBM Middleware as first-class citizens in my containerized applications. And if I can do it, you can too!
I hope you find this sample helpful. I appreciate your time and attention and look forward to your feedback!