Cloudbench (CBTOOL)

Cloudbench Tool (CBTOOL) is a multi-benchmark harness that automates infrastructure as a service (IaaS) cloud stress and scalability testing through running controlled experiments.

Started in late 2009 as a framework to measure performance application on a cloud (specifically memory overcommit in Kernel-based virtual machines), the harness evolved into a versatile tool for measuring cloud performance in multiple dimensions.

These characteristics lead it to be adopted as the harness that drives SPEC Cloud™ IaaS 2016, the first industry-standard cloud benchmark. Cloudbench is routinely updated and expanded. It supports more than 65 workloads or workload profiles in multi-instance configurations and more than ten cloud providers by default, such as Amazon EC2, Digital Ocean, Google Compute Engine, and IBM Softlayer. The workload instances can be containers, virtual machines, or even bare metal.

After evaluating the experience of multiple research and development cycles in many different cloud IaaS infrastructures in the past six years, it was clear that simply performing functional unit or synthetic testing on its multiple sub-components (for example, CPU performance or network bandwidth) did not yield realistic results on the behavior of a cloud. Cloudbench creates experimental scenarios where multiple, reasonably complex, and realistic workloads are deployed and monitored under strict control, against multiple different IaaS clouds.

Experimentors can use a declarative language with primitives such as “Virtual Application Instance” (VApp or AI) to design multi-instance workloads, and “Application Instance Deployment Request Submitter” (AIDRS). They can create a scenario where the workload population mix, workload resource requirements, workload arrival rate and residency time, and the number of simultaneous workload creation requests, are varied through simple configurations.

Why should I contribute?

Are you working on a new IaaS cloud not yet covered by CBTOOL? Do you have a new cool workload that you want to deploy at scale in a cloud environment as part of a test? Given the status of CBTOOL as the engine behind SPEC Cloud™ IaaS 2016, expansions to this standard benchmark are primarily done from the workloads already supported. We gladly accept contributions for both “cloud adapters” and “workload types”, which benefit the whole Cloud Performance Evaluation community by increasing the repeatability and standardization.

What technology problem will I help solve?

You will save time. Cloudbench saves a cloud architect or developer time by automating large and complex experimental scenarios to uncover problems and (performance) regressions due to continuous changes.

CBTOOL is a framework, where the library of cloud adapters (code that allows the experimental directives to be translated into an IaaS cloud native requests) and workload types (the scripts that control the deployment and execution of a workload, internally called “Virtual Application Types”) are constantly expanded.

How will CBTOOL help my business?

You might be a cloud customer interested in determining the performance of various clouds. Or you might be a public or private cloud provider looking to find and replicate performance issues that only occur at scale. CBTOOL can help you answer questions that guide your cloud selection and help you find those nasty performance bugs in your cloud before they reach customers.