Measuring performance with the Daytrader 3 benchmark sample

The Daytrader 3 benchmark sample provides a suite of test cases to measure scalability and performance. This article explains how to run a web-based workload driver (Apache JMeter) against it to measure throughput performance. Also, some IBM DB2 tuning tips.

What is the DayTrader 3 benchmark sample?

The DayTrader 3 benchmark sample provides a suite of workloads for characterizing performance of a Java EE 6 Application Server. The workloads consist of an end-to-end Web application and a full set of Web primitives. Together, the DayTrader 3 application and Web primitives provide versatile and portable test cases that are designed to measure aspects of scalability and performance.

The DayTrader 3 benchmark sample is a Java EE 6 application built around an online stock trading system. The application allows users to login, view their portfolio, look up stock quotes, buy and sell stock shares, and more.

DayTrader 3 is built on a core set of Java EE 6 Web Profile technologies and a few Java EE 6 Full Profile Technologies.

Web profile technologies:

• Servlet 3.0
• JSP 2.2
• JSF 2.0 (new in DayTrader 3)
• JPA 2.0
• EJB 3.1
• JDBC 4.0

Full profile technologies:

• JMS 1.1
• Message-Driven Beans
• JAX-RS 1.1 (new in DayTrader 3)

There are two main modes we use for benchmark testing. The mode can be set on the configuration tab of the application, along with many others settings:

1. EJB3 Mode: Uses EJBs with JPA to connect to the database,
2. Direct (JDBC) Mode: Uses JDBC to connect directly to the database.

The following diagram provides a high-level overview of the full workload application architecture:

How to obtain the DayTrader 3 Benchmark Sample

There are two ways to obtain the DayTrader 3 Application:

• Liberty DayTrader3 sample
  • This download includes a pre-configured Liberty 8.5.5.X server with a simple Apache Derby database.
• WebSphere Performance Page (under Downloads on the right side of the page)
  • This download contains instructions to install and configure the application on WebSphere Application Server Full profile and the Liberty profile. It contains instructions for using different databases including IBM DB2, Apache Derby, and Oracle Database.
  • Also included is an Apache JMeter script and properties file that can be used for performance testing (see below).
  • The download contains two versions of the DayTrader 3 application. One has IBM specific functionality (Distributed Map Cache) and one has no IBM specific function, and can be installed on other Application Servers.

How to run Apache JMeter against the an Application Server with the DayTrader 3 application installed.

1. Download and unzip Apache JMeter onto a separate machine that will be used to drive the load to the application server.
2. Copy daytrader3.jmx and jmeter.properties from the download on the WebSphere Performance Page to APACHE_JMETER_HOME/bin. The files can be found in the zip file at DayTrader3Install/DayTrader3-EE6/Apache_JMeter_script. The README.txt in the same location has documentation on additional options for JMeter.
3. Make sure the application is installed correctly and is running.
4. Start JMeter (I personally prefer non-GUI mode.):

   cd APACHE_JMETER_HOME/bin
   ./jmeter -n -t daytrader3.jmx -JHOST=myserver -JDURATION=180
   

I usually do three 180 second warm up runs and then three 180 second measurement runs. I also reset the database between each run, which can be done on the configuration tab of the application.

Example output:

summary +  35432 in     6s = 5890.6/s Avg:     8 Min:     1 Max:    54 Err:     0 (0.00%) Active: 50 Started: 50 Finished: 0
summary = 1041929 in   174s = 5985.9/s Avg:     8 Min:     1 Max:    85 Err:     0 (0.00%)
summary +  32588 in   5.5s = 5968.5/s Avg:     8 Min:     1 Max:    53 Err:     0 (0.00%) Active: 0 Started: 50 Finished: 50
summary = 1074517 in   180s = 5986.0/s Avg:     8 Min:     1 Max:    85 Err:     0 (0.00%)

In the example above, I was able to achieve 5,986 requests/second with an average response time of 8ms.

This measurement was done using the following hardware and software:

• Liberty 8.5.5.0 running with IBM JDK 7 SR4 64bit
• 4-cores of an Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz processor
• SUSE Linux Enterprise Server 11.3 with 2MB Large Pages
• JDK options: -Xms1024m -Xmx1024m -Xcompressedrefs.

By default, the daytrader3.jmx script stresses the application like so:

• 85% – Servlet/JSP Requests
  • 30% – Quote Request
  • 20% – Home Page Request
  • 10% – Account Page Request
  • 10% – Portfolio Page Request
  • 4% – Buy Request
  • 4% – Sell Request
  • 2% – Update Profile Request
  • 1% – Register Request
  • 4% – Logout Request
• 10% – JSF Requests
  • 7% – Quote Request
  • 3% – Account Request
• 5% – JAX-RS Requests

Tips for running with IBM DB2

I have found that applying the following DB2 tunings, after creating the database, is helpful with allowing the application to scale better. This assumes the database is running on a Linux system. After I apply the tunings, I then backup the database and restore it before each benchmark run.

To configure and back up the database:

DB=tradedb
mkdir -p ~/backups/${DB}
db2 update dbm cfg using notifylevel 0
db2 update dbm cfg using diaglevel 1
db2 update dbm cfg using NUM_POOLAGENTS 500 automatic MAX_COORDAGENTS 500 automatic MAX_CONNECTIONS 500 automatic
db2 -v update db cfg for ${DB} using MAXLOCKS 100 LOCKLIST 100000
db2 connect to ${DB}
db2 update db cfg for ${DB} using maxappls 500 automatic
db2 update db cfg for ${DB} using logfilsiz 8000
db2 update db cfg for ${DB} using logprimary 32
db2 update db cfg for ${DB} using dft_queryopt 0
db2 update db cfg for ${DB} using softmax 3000
db2 update db cfg for ${DB} using chngpgs_thresh 99
db2 -v alter bufferpool IBMDEFAULTBP size -1
db2 -v connect reset
db2 -v update db cfg for ${DB} using BUFFPAGE 262144
db2set DB2_KEEPTABLELOCK=CONNECTION
db2set DB2_USE_ALTERNATE_PAGE_CLEANING=ON
db2set DB2_MINIMIZE_LISTPREFETCH=YES
db2 connect reset
db2 terminate
db2stop force
db2start
db2 connect to ${DB}
db2 reorgchk update statistics
db2 connect reset
db2 terminate
db2 backup db tradedb to ~/backups/${DB}

Restore Database

db2stop force
db2start
db2 restore db tradedb from ~/backups/tradedb replace existing