Overview

Skill Level: Beginner

Test Metrics help you evaluate the performance of your testing software in the cloud infrastructure. This article explains everything that you need to know.

Ingredients

Cloud computing offers a virtually unlimited pool of resources that you can use to develop, deploy, test and monitor your applications. However, the reason why the cloud has stayed popular is that it's 100% available, reliable and scalable—the fundamental characteristics of the cloud. When cloud came out, organizations were eager to replicate these cloud characteristics for testing software. There was a growing need to use the cloud computing strategies to test the cloud architecture and the applications running on it. With it also emerged the need to evaluate the quality of the cloud testing services to ensure that they are efficient and effective. This article discusses the different metrics that you should be aware of while running tests in the cloud.

 

 

Step-by-step

  1. Understanding Cloud Testing and Test Metrics

    Performance testing, load testing, and stress testing are some of the common test scenarios covered by the cloud. The cloud can easily simulate the geographically distinct user traffic and meet the hardware requirements to run the tests which would otherwise incur high cost on traditional testing systems. To analyze the effectiveness of the cloud testing infrastructure, we need to have a quantitative measure in place. Test metrics are particularly useful in that regard because they help you to estimate the progress and the quality of a test scenario.

    There are many factors that you need to consider while running cloud testing services. This includes bandwidth, storage, response time and throughput, volume and utilization etc. This article takes you through different metrics that you have to consider to optimize your test efforts, namely, performance and load testing in the cloud. 

  2. Test Metrics for Performance & Load Testing

    Performance is one of the critical factors to consider while testing a cloud application as it can directly impact the user experience. This test practice is usually done to measure certain performance attributes such as throughput, responsiveness, bottleneck, limitations, and latency when the application is under different workloads. The cloud can simulate traffic on-demand to create a workload that matches the performance test profile.

    Here are some of the key measurements that you should consider while testing the performance of an application.

    1. CPU performance metrics
    2. Response metrics
    3. Volume metrics
    4. Miscellaneous performance metrics
  3. CPU Performance metrics

    Tests that capture the performance of the  CPU hotspots are always helpful to make useful decisions. For instance, in a traditional PHP server, you should be looking at the count of busy/idle worker threads. For a server running node.js, monitoring the CPU usage should give you sufficient information if the server is taking an unnecessary synchronous load. If the value of this metric exceeds a threshold value, it’s high time that you add more servers or spawn a new thread for the CPU intensive task.

  4. Response Metrics

    Response times are critical metrics that evaluate the performance of the application from the user’s perspective. This includes response time, page load time, error rate etc. Response time is the time taken by a server to return a response back to a user request. Based on the request-response cycle time, it is further classified into average response time, peak response time, time for the first byte and time for the last byte, etc. Page load time is the time taken by the application to process all the responses and render the view. Error rate measures the number of errors encountered under load.

  5. Volume Metrics

    Volume metrics are measurements that describe how well the application handles the load and the different attributes that you have to consider from the server’s point of view. The key volume metrics are—request throughput, requests per second (RPS), concurrent users and resource utilization. Throughput describes the number of transactions that the server can take in any given amount of time. The RPS metric gives you information about how many requests are made from the front-end of the application to the server. The number of concurrent users at any given time gives an approximate estimate of the maximum load under which the application can work without any repercussions.

  6. Other Performance Metrics

    There are many other performance test metrics that you might come across that are relevant to your project. Here are a few that I feel are important:

    • Bandwidth and Network requirements. Bandwidth is often the limiting factor that prevents an application from performing as expected on heavy load. 
    • Memory issues such as memory leaks. Many things could go wrong with memory — the way it’s configured or bad code that leaks memory. 
    • Process level metrics. Sometimes, key processes need to be run in a sandboxed environment so that other processes don’t eat up its resources.
    • CPU utilization. Optimized utilization of the resources is also important. Keeping certain nodes busy and rest of the nodes idle results bad resource utilization. 
  7. Summary

    If you’ve been running tests in the cloud, you should already know that some of these are critically important to evaluate the efficiency of the cloud testing infrastructure in place. You can find more details in this article. Some of the test metrics might be case specific whereas others are not. If you have any thoughts to share about cloud testing in general or test metrics for cloud testing, let us know in the comments.

Join The Discussion