Monitor your website using JMeter

Introduction

When you hear the term “watchdog,” what does that conjure up in your mind? Something meant to guard, protect, or notify? Something that barks, snarls, or yells? Something with alarms, sirens, or blow horns? Depending on the context, your mind’s eye image may be spot on with those descriptions, but in this case, let’s get a bit more technical with a different kind of watchdog: JMeter.

In this tutorial, we introduce you to Apache JMeter, “a 100% pure Java application designed to load test functional behavior and measure performance.” Learn how to easily configure JMeter to monitor an application for various events so that you can identity problems quickly and fix them before they affect your end users.

Keeping web sites readily available and healthy is a challenge. Using the DevOps practice of continuous integration begs the complimentary use of test automation to ensure code deliverable quality. Using JMeter in the testing domain not only covers function and performance, but also monitoring and notification capabilities when failures are encountered.

There are other test tools to choose from and each environment will have its requirements. Using the example JMeter test plan provided (based on a production level custom test plan), along with gained knowledge, we feel it’s both simple and straightforward to use compared to other test tools. Using JMeter and techniques shown, we’ve developed around 200 test cases running against four environments and associated systems/services every hour for years. Having this testing in place for our suite of applications, along with email notifications and results reporting, provides a level of confidence that is priceless.

Prerequisites

To use JMeter, you need to do the following:

  • Install a fully compliant JDK. We suggest you install the latest minor version of a major version; for example, JDK 8.
  • (Optional) On Windows, include the JDK executable directory in the environment variable PATH (i.e., C:\Program Files\IBM\Java80\bin).
  • (Optional) Set the environment variable JAVA_HOME to match the JDK executable directory.
  • Download the mockSites.zip file to follow along with this tutorial.
  • Download the Watchdog.jmx file.

Note: If you run into any path-related issues in subsequent steps, you probably want to ensure these optional prerequisites are complete.

Steps

Follow the steps in this section to get up and running with JMeter. For detailed information, visit Apache JMeter Getting Started.

  1. Install JMeter
  2. Start the JMeter application
  3. Set up mock websites for testing
  4. Get an introduction to the JMeter test plan
  5. Configure and execute a JMeter test plan
  6. Simulate failures to see how JMeter behaves
  7. Understand how the test plan works

Step 1. Install JMeter

Once you’ve completed the prerequisites, you can install the JMeter software:

  • Download the Apache JMeter application software.
  • To install the release build, unzip the zip/tar file into the directory where you want to install JMeter. Provided that you have a JRE/JDK correctly installed and resolvable, there is nothing more to do.

Step 2. Start the JMeter application

To start the JMeter application, launch one of the following commands:

  • Windows: Use jmeter.bat
  • Linux/UNIX: Use jmeter

The respective command file is found in the application’s bin directory. Once launched, after a short time, the JMeter (GUI) application should appear.

View of the JMeter GUI

Step 3. Set up mock websites for testing

For the purposes of this tutorial, you will create a mock website to execute tests against. The mock site serves as the target application for the provided sample JMeter test plan. We will simulate an outage and other problems and show you how to use JMeter to identify and monitor these issues. For this purpose, multiple (mock) web applications will be created using Node.js.

Install Node.js

Node.js is a popular asynchronous event-driven JavaScript runtime used to build scalable network applications. In this tutorial, Node.js provides the infrastructure and custom-developed code for the mock websites. You need to install Node.js, its package manager (npm), and http-server, a simple command-line HTTP server.

To install, visit Node.js, choose your version, and run the installation package to install the Node.js runtime. On Windows, the installation path is most likely C:\Program Files\nodejs

Install npm and package http-server

npm is the package manager for the Node.js JavaScript platform. It puts modules in place so that node can find them and manages dependency conflicts intelligently.

http-server is a simple, zero-configuration command-line http server. It is powerful enough for production usage, but it’s simple and hackable enough to be used for testing, local development, and learning.

npm and http-server provide the artifacts needed to create the mock websites that we need JMeter to test.

Use the Node.js command prompt to install the package (globally) with the npm command:

npm install --global http-server

Optionally, validate the Node.js installation using the ‘hello world’ example.

Install the mock websites

Source code for the mock websites is provided. Download and extract the necessary artifacts. We recommend that you set up a projects directory to hold the source code. An example project directory structure might look like the following.

Project directory for same websites with folders named apple, banana, pomelo

Access the mock websites

You can start the provided mock sites collectively or individually. Below are steps for both ways.

To start them all at once:

  1. Launch the Node.js command prompt.
  2. Change to the directory containing the mock site resources.
  3. Launch command startMockSites.cmd. All three instances of the mock sites will start.

Picture showing launch of the Node.js command prompt

To start each site individually, use the following steps.

Launch the sample Apple website:

  1. Launch the Node.js command prompt.
  2. Change to the directory of the Apple website.
  3. Run the http-server command.
  4. Access the Apple website at http://localhost:8080.

Output of running starting the Apple site via the http server command

Launch the sample Banana website:

  1. Launch the Node.js command prompt.
  2. Change to the directory of the Banana site.
  3. Run the http-server command.
  4. Access the Banana site at http://localhost:8081.

Launch the sample Pomelo website:

  1. Launch the Node.js command prompt.
  2. Change to the directory of the Pomelo site.
  3. Run the http-server command.
  4. Access the Pomelo site at http://localhost:8082.

Terminate the http-server command for a site by pressing Ctrl+C from within the command prompt window for that site.

Step 4. Get an introduction to the JMeter test plan

Check out the sample JMeter test plan. The sample contains tests that we will show you how to execute against the mock sites so that you’ll get a feel for how JMeter works. Have the sample test plan ready for testing.

Test plan elements

This section gives you a high-level understanding of the various elements of a JMeter test plan. If you want more details about these elements, refer to the JMeter web site.

  • Thread group elements are the beginning points of a test plan. All controllers and samplers must be under a thread group. You can create and execute multiple thread groups in parallel. In the example provided, variables are used to control when a test should be performed and at what intervals.

There are two types of controllers that drive the processing of a test in JMeter: samplers and logical controllers.

  • Samplers tell JMeter to send requests to a server.
  • Logical controllers customize the logic that JMeter uses to decide when to send requests.
  • Test fragment elements are special types of controllers that exist at the same level as the thread group element. Test fragment elements are different from a thread group because they aren’t executed unless referenced by either a module controller or an include controller.
  • Listeners provide access to the information JMeter gathers about the test cases while JMeter runs. The “View Results Tree” listener shows details of sampler requests and responses and can display basic HTML and XML representations of the response. Other listeners provide summary or aggregation information.
  • Assertions allow you to assert facts about responses received from the server being tested. Using an assertion, you can essentially “test” that your application is returning the results you expect it to.
  • Configuration elements work closely with a sampler. Although it does not send requests (except for HTTP(S) Test Script Recorder), it can add to or modify requests.
  • Pre-processor executes an action prior to a sampler request being made. If a pre-processor is attached to a sampler element, then it executes just prior to that sampler element running. A pre-processor is most often used to modify the settings of a sample request just before it runs or to update variables that aren’t extracted from response text.
  • Post-processor executes an action after a sampler request has been made. If a post-processor is attached to a sampler element, then it will execute just after that sampler element runs. A post-processor is most often used to process the response data, often to extract values from it.
  • Properties are defined in jmeter.properties, are global to JMeter, and are mostly used to define default JMeter uses.
  • Variables don’t have to vary. You can define them once and they will not change value unless you change them. You can use variables as short-hand for expressions that appear frequently in a test plan or for items which are constant during a run, but which may vary between runs. In this example, the name of a host and the email content are stored in variables.
  • Scoping rules are critical to a test plan. The JMeter test tree contains elements that are both hierarchical and ordered. Some elements in the test trees are strictly hierarchical (listeners, config elements, post-processors, pre-processors, assertions, timers), and some are primarily ordered (controllers, samplers). When you create your test plan, you will create an ordered list of sample request (via samplers) that represent a set of steps to be executed. These requests are often organized within controllers that are also ordered.

Step 5. Configure and execute a JMeter test plan

Before running a JMeter test plan, you need to configure it to the environment being tested. To modify the configuration for the provided sample test plan (Watchdog.jmx), open the plan in the JMeter application GUI. From the main menu area, click File > Open and select the test plan to open.

Set the user-defined variables.

To ensure the proper interested parties receive notifications, you need to configure some variables:

  1. Specify the SMTP server from which mail is processed.

    • USER_SMTP_SERVER: host name or IP address of the SMTP server.
  2. Configure the mail sender and recipients:

    • USER_REPORT_SEND_FROM: sender in the mail of test result
    • USER_RESULT_SEND_TO: email addresses to be notified when test result (separate email address with semicolon)
    • USER_RESULT_FAILED_COPY_TO: email addresses to be notified in copy list when test result is FAILED (separate email address with semicolon)

JMeter GUI showing user-defined variables

Test plan execution

  1. Make sure the mock sites are running, then open JMeter (using the sample test plan).
  2. Once JMeter is open and available, click the Start button.
  3. Click the listener View Results Tree to see any test results.
  4. Check your mail inbox for three emails containing test report information.

Image of user highlighting View Results Tree

Test plan results

The subject line of the test report contains the status, count of failed test cases, which environment was tested, when the test was performed, and the elapsed time of the test. The test report also contains the following:

  • Host name and IP address of target environment
  • Host name and IP address of JMeter instance
  • Time the test performed
  • Elapsed time of the test
  • FAILED test cases and count
  • PASSED test cases and count
  • Test parameters
  • Details for failed cases

Email of test plan results

Step 6. Simulate failures to see how JMeter behaves

Now let’s see how JMeter behaves in failure situations.

Simulate the outage (heartbeat not available)

To avoid failed test cases flooding the test report, the heartbeat verification is separated from the regular tests. The HEARTBEAT_NOT_AVAILABLE notification is configured to be sent every hour, but target site availability is checked every minute. In this example, page monitor.html is the heartbeat of the application.

Simulate the outage by doing the following:

  1. Terminate the banana instance of http-server by pressing Ctrl+C.
  2. Go to JMeter.
  3. Stop and then restart the test plan. The HEARTBEAT_NOT_AVAILABLE notice will be sent.
  4. Go to your mail box and look for the HEARTBEAT_NOT_AVAILABLE notice.

The following image shows the termination of the sample websites:

Image showing the termination of the sample websites

The following image shows a “HEARTBEAT NOT AVAILABLE” notice:

Image of notice of HEARTBEAT NOT AVAILABLE notice

While JMeter continues to run, it checks the site availability every minute.

Try to bring up the mock Banana website by following these steps:

  1. Launch the Node.js command prompt.
  2. Change to the directory of site banana.
  3. Run the http-server server.
  4. You may access the site banana on http://localhost:8081.
  5. Go to JMeter and click the listener View Results Tree. Note that the test on banana is performed in the next minute.
  6. Go to your mail box and then check SUCCESS test result for banana.

Simulate the problem (assertion failure)

You may simulate the assertion failure by changing the file name of web page in the mock site:

  1. Go to the directory of the banana site.
  2. Rename the about.html file name to aboutFail.html.
  3. Stop and restart the test plan. The FAILED notice will be sent for the respective instances.

The following image shows the website resources after changing the file name:

View of web site resources after changing file name

The following image shows the failure notice for test results:

Failure notice for test results

Step 7. Understand how the test plan works

Now that you’ve seen both success and failure scenarios using the sample JMeter test plan, let’s look at some key points to explain on how it really works.

Major Thread Group

The major Thread Group is defined to run forever and includes both production and test environments. The target applications are not necessarily tested continuously. This means that delays are often used for test iterations using a time interval. In the major Thread Group, there is a Test Action with a default pause duration of 60 seconds defined with variable DELAY_BEFORE_NEXT_ITERATION_MILLISECONDS. This provides a pause or delay until the next test iteration begins against the target systems.

View of the test delay

Test the target systems in the major thread group

Our example includes three target websites to be tested, grouped by production and test environments. The variables TEST_TARGET_SYSTEM, TEST_TARGET_PORT, TEST_TARGET_PROTOCOL, and TEST_TARGET_SYSTEM_IS_PRODUCTION are altered before running the test cases. The following module controller runs the test on ${TEST_TARGET_SYSTEM} when the next test schedule is due.

Run the test when next schedule for the target system is due

In this module controller, the first sampler tests if the current time is greater than the next scheduled time (TIME_NEXT_TEST_XXX) of the target system that is due for testing. The variable RUN_TEST_IN_THIS_ITERATION is set as true and the descendant of the following ‘if’ controller will be run.

Prepare for the test

  1. Clear the test results for any target systems. This also clears the JMeterTreeNode in listener View Result Tree. Clearing the test results when using the JMeter GUI is important as a large amount of system resources are consumed when running tests.
  2. Get the JMeter machine IP Address. This gets the IP address that will be included in the test report. It will be helpful to distinguish which JMeter instance generates the test report.
  3. Reset variables. This resets all the variables for the test report.

View of test preparation actions

Check the availability of the target system

Before running tests against the target system, you should verify the availability of the target system. The first sampler tests the simple web page monitor.html to determine if the target system responds properly.

The HEARTBEAT_NOT_AVAILABLE notification is sent if the target system is not available. The TEST_INTERVAL_MILLISECONDS variable controls the schedule interval determining when to send the next HEARTBEAT_NOT_AVAILABLE notification. These samplers do not change the TIME_NEXT_TEST_XXX, and the next schedule to test the target system is still due.

View of target site availability sampler

Run test cases for the target system

Finally, the test cases (samplers) are executed. The assertion can be appended as a post-processor action to verify the response.

How the report is constructed

Using a pre-processor script before each test is the secret of how the report is constructed. In the pre-processor above, the script does the following:

  1. Resets variables for the test report if the variable is null
  2. Appends the test result of previous sampler to the test result
  3. Groups the test results as PASSED and FAILED test cases
  4. Appends the details (Sample label, Sample URL, Response data) for FAILED test cases

Since the test result of all samplers will be included, the prefixes are defined and used as a way to ignore or treat a failed result as PASSED. The prefixes are as follows:

  1. OPTIONAL – treat as pass always, but put into log
  2. TEST_OPTIONAL – treat as pass in non-production environments
  3. WEEKEND_OPTIONAL – treat as pass in weekend
  4. PREPARE – ignore the sampler, and exclude from the test result
  5. KNOWN BUG – ignore the failure of the sampler

Image highlighting the code that constructs the report (Script before each test)

Send the report via an email notification

The Mailer Visualizer listener in JMeter facilitates sending test results; our example uses the SMTP Sampler. It sends an email based on or using the following information:

  1. Email is sent to recipient USER_RESULT_SEND_TO and USER_RESULT_FAILED_COPY_TO.
  2. Message content is generated using a predefined template with variables of test result.
  3. Subject contains the overall test status SUCCESS or FAILED with count of failed test cases.
  4. The machine host name of the JMeter instance is included to help determine where the email notification came from.
  5. The file name of the test plan is included.
  6. The FAILED test cases are listed.
  7. The PASSED test cases are listed.
  8. The test parameters are included.
  9. Details of FAILED test cases are included in the bottom of the email.

Tips

Send email to debug.mail.address instead of actual recipients

The type of testing preformed will depend on the type and amount of configuration needed. JMeter is highly configurable per its various properties files. One such JMeter properties file is users.properties. For details, visit Apache JMeter Properties Reference.

A particular property that is used in the example provided is debug.mail.address, which can be defined in the user.properties file. It’s recommended to initially assign yourself (as tester) to this property. Any test result notifications will be sent to assigned users of this property instead of the ‘Recipients’ variables. This is a preferred approach when initially developing a test plan as it overrides and prevents having to modify or use the ‘Recipients’ variables, which may ultimately contain a different list of users. Locate the properties file (JMeter bin directory) and use a favorite editor to make recommended changes.

You can define additional JMeter properties in the file defined by the JMeter property parameter attribute user.properties which has the default value user.properties. The file will be automatically loaded if it is found in the current directory or if it is found in the JMeter executable directory.

Launch JMeter using CLI mode

For load testing, you must run JMeter using command line interface (CLI) this mode (non-GUI) to get the optimal results. To do so, use the following command options:

-n

This specifies JMeter is to run in CLI mode:

-t

Name of JMX file that contains the Test Plan:

jmeter -n -t Watchdog.jmx

Summary

By now, hopefully you’ve gained an appreciation of Apache JMeter’s capabilities, a better working knowledge of the test tool, as well as ideas on how you might inject JMeter into your DevOps practices. JMeter might not literally bark, snarl, or blow a horn, but it can certainly notify you of application issues and serve as a watchdog that you can count on. Sound the alarm and give JMeter a try!