Tests are run using a simple two-container setup: a client container connected to a server container using IPv4 over Docker’s virtual networking capabilities. The two containers are executing on the same physical host and their lifecycles are managed and synchronized externally through scripting facilities provided by the host operating system.
The client container runs PerfHarness as a standalone Java application using the Java Virtual Machine shipped with the product under evaluation. Apart from this difference, the two products are tested using identical client configurations for each combination of test parameters. We place no artificial restrictions on resource usage by the client container and aim to run in a sufficiently overprovisioned environment so that the underlying hardware resources are never exhausted.
The server container runs the product under test in a configuration suitable for the test case and combination of parameters being evaluated. The suitable configuration will depend on the product as well as the test case and parameters; certain configuration options may be available in one product but not in the other. We use equivalent configurations where possible. We further rely on Docker’s resource limiting capabilities to restrict CPU usage by the server, but impose no other resource constraints. We aim to keep the resource usage of the server well below the physical host’s limits.
Both client and server containers are created from pre-built images containing the appropriate client or server product versions. We use separate images for IBM Integration Bus Version 10.0.0.16 server, its corresponding test client, and IBM App Connect Enterprise Version 188.8.131.52 server and its corresponding test client. Containers are kept only for the duration of the test run; they are discarded afterwards and new ones are created for subsequent runs.
Although a comparison of the respective performance characteristics of running the products in containers against those of running them on bare metal hardware is outside the scope of this document, we anticipate that the use of containers presents a small, possibly negligible overhead on the CPU usage figures that will likely not be reflected in our measurements, and that there may be a larger overhead associated with performing I/O operations such as the network activity associated with transferring messages over HTTP . We expect that the products running inside containers will generally not perform better than the same products running on bare metal hardware.
 Felter, W., Ferreira, A., Rajamony, R., Rubio, J. An Updated Performance Comparison of Virtual Machines and Linux Containers. Technical Report RC25482 (AUS1407-001), July 21, 2014. IBM Research Division, Austin, TX.