Understanding Cloud Testing and Test Metrics
Performance testing, load testing, and stress testing are some of the common test scenarios covered by the cloud. The cloud can easily simulate the geographically distinct user traffic and meet the hardware requirements to run the tests which would otherwise incur high cost on traditional testing systems. To analyze the effectiveness of the cloud testing infrastructure, we need to have a quantitative measure in place. Test metrics are particularly useful in that regard because they help you to estimate the progress and the quality of a test scenario.
There are manyÂ factors that you need to consider while running cloud testing services. This includes bandwidth, storage, response time and throughput, volume and utilization etc. This article takes you through different metrics that you have to consider to optimize your test efforts, namely, performance and load testing in the cloud.Â
Test Metrics for Performance & Load Testing
Performance is one of the critical factors to considerÂ while testing a cloud application as it can directly impact the user experience. This test practice is usually done to measure certain performance attributes such as throughput, responsiveness, bottleneck, limitations, and latency when the application is under different workloads. The cloud can simulate traffic on-demand to create a workload that matches the performance test profile.
Here are some of the key measurements that you should consider while testing the performance of an application.
- CPU performance metrics
- Response metrics
- Volume metrics
- Miscellaneous performance metrics
CPU Performance metrics
Tests that capture the performance of theÂ CPU hotspots are always helpful to make useful decisions. For instance, in a traditional PHP server, you should be looking at the count of busy/idle worker threads. For a server running node.js, monitoring the CPU usage should give you sufficient information if the server is taking an unnecessary synchronous load. If the value of this metric exceeds a threshold value, itâ€™s high time that you add more servers or spawn a new thread for the CPU intensive task.
ResponseÂ times are critical metrics that evaluate the performance of the application from the userâ€™s perspective. This includes response time, page load time, error rate etc. Response time is the time taken by a server to return a response back toÂ a user request. Based on the request-response cycle time, it is further classified into average response time, peak response time, time for the first byte and time for the last byte, etc. Page load time is the time taken by the application to process all the responses and render the view. Error rate measures the number of errors encountered under load.
Volume metrics are measurements that describe how well the application handles the load and the different attributes that you have to consider from the serverâ€™s point of view. The key volume metrics areâ€”request throughput, requests per second (RPS), concurrent users and resource utilization. Throughput describes the number of transactions that the server can take in any given amount of time. The RPS metric gives you information about how many requests are made from the front-end of the application to the server. The number of concurrent users at any given time gives an approximate estimate of the maximum load under which the application can work without any repercussions.
Other Performance Metrics
There are many other performance test metrics that you might come across that are relevant to your project. Here are a few that I feel are important:
- Bandwidth and Network requirements.Â Bandwidth is often the limiting factor that prevents an application from performing as expected on heavy load.Â
- Memory issues such as memory leaks. Many things could go wrong with memory — the way itâ€™s configured or bad code that leaks memory.Â
- Process level metrics. Sometimes, key processes need to be run in a sandboxed environment so that other processes donâ€™t eat up its resources.
- CPU utilization. Optimized utilization of the resources is also important. Keeping certain nodes busy and rest of the nodes idle results bad resource utilization.Â
If youâ€™ve been running tests in the cloud, you should already know that some of these are critically important to evaluate the efficiency of the cloud testing infrastructure in place. You can find more details in thisÂ article. Some of the test metrics might be case specific whereas others are not. If you have any thoughts to share about cloud testing in general or test metrics for cloud testing, let us know in the comments.