We have designed a number of test cases, described below, to evaluate the performance of the various functional sub-units of the products under test. We perform all test runs in 3 different concurrency configurations, using 3 different message sizes in each (9 configurations altogether for each test scenario):
- 1, 2, and 4 virtual CPU cores constrained by Docker’s resource limiting capabilities.
- 2k, 20k, and 200k request messages in the format appropriate for the test case, XML or JSON. The message sizes represent the size of the input messages in the XML format. As the JSON input messages essentially represent the same data as their XML counterparts, they are significantly smaller (approximately 1.3k, 13k, and 130k). Nevertheless, for the sake of simplicity, we will refer to them using the XML message sizes. We also note that we use a different set of XML messages for SOAP interactions than for raw XML over HTTP and that the smallest SOAP message is closer to 4k in size.
Furthermore, all test runs are subject to the following parameters:
- Test runs are 150 seconds long.
- The client sends at most 5000 requests per persistent HTTP connection.
- The client runs 4 times as many client threads (and hence, simultaneous persistent HTTP connections) as the number of virtual server cores. We set the number of flow instances on the server to 3 times the number of virtual server cores.
- In IBM Integration Bus Version 10.0.0.16, we create a single Integration Node with a single Integration Server, and deploy a pre-packaged BAR file containing a single message flow with its supporting artifacts. In App Connect Enterprise Version 220.127.116.11, we deploy a similar BAR file to a Standalone Integration Server for the purpose of the test. Although we test the two products using the same set of deployable artifacts, we rebuild each of them using the Integration Toolkit that ships with the product.
E.g., for a test run using 4 cores, PerfHarness will run 16 client threads, each maintaining and recycling as appropriate a single persistent HTTP connection throughout the duration of the test run, against a server running 12 message flow instances (Additional Instances property set to 11).