Understanding Application Resource Management using Turbonomic

Continuously assuring the performance of applications is perhaps one of the most critical responsibilities of a Site Reliability Engineer (SRE). Loss of performance is detrimental in modern businesses. A slow-performing application can drive prospective clients to competitors. This loss of business has happened and continues to happen every day in environments of all scales and architectures, especially as we see more advanced applications driving the business behind retail and other companies. And, this issue becomes even more complex and untenable during peak and holiday seasons, during promotional deals, or during similar types of activities that involve bursts of user activities. Non-performing applications can directly result in loss of customer satisfaction, which results in the erosion of your customer base and consequently revenue loss.

Typically, enterprise-class applications are hosted in modern private virtualized data centers, public cloud data centers, or across these hybrid environments that encompass both on-prem and public cloud infrastructures. The applications can be traditional, containerized, or a mixture of both. It can be assumed that enterprise applications are reasonably well tested before they are deployed in production and that production environments are multi-tenanted. Most of the time, well-behaved applications manifest performance issues due to congestion of one or more resources, such as CPU, memory, storage, and network resources.

Assuring application performance for well-designed applications that reside in multi-tenant environments, which host many applications with various service level objectives (SLOs), is downright daunting. Traditionally, SREs or developers can attempt to address the application performance issues by judiciously watching relevant metrics from application performance monitoring (APM) tools like Instana or Dynatrace and adjusting the available resources. The manual attempt is complex and might require detailed knowledge of the applications. And, using APM tools can be error-prone and, most concerningly, is purely reactive in nature: when the SRE has detected the performance issues, the applications have already slowed down.

To avoid performance issues, infrastructure architects often resort to over-provisioning resources. After a typical sizing exercise, architects multiply the resource requirements by 1.x times before provisioning the hosting environments for deployments. The value of x can often be as high as 25. While architects are attempting to avoid application performance issues proactively, it is wasteful of resources. In a public cloud environment, where resources are “rented,” extra provisioning can make businesses pay for underutilized or unused resources all the time.

To address these application performance issues, SREs or developers need an application resource management (ARM) utility. Ideally, an ARM utility should continuously monitor the environment, collect metrics from various abstraction layers, and provide actionable, preventative recommendations to mitigate performance risk to assure that their applications can meet their SLOs. The use of a good ARM utility should not demand deep application, infrastructure, or APM tool knowledge from its users.

Turbonomic is just such an ARM utility. Almost all the recommendations that Turbonomic generates are automatable. Turbonomic can provide scaling up recommendations to acquire one or more resources after it verifies that the underlying infrastructure can safely deliver them. It can also recommend scaling down actions to consolidate workloads and return unused resources to the provider.

Although Turbonomic started as an ARM utility for traditional applications deployed in virtualized on-prem infrastructures, it has successfully extended itself to cover the infrastructures provided by public clouds like Amazon Web Service (AWS), Microsoft Azure, and Google Cloud Platform (GCP). (In this article, unless otherwise mentioned, the on-prem environment stands for a hypervisor or a hyper-converged infrastructure, and the cloud stands for public cloud infrastructure provided by AWS, Azure, or GCP.)

For cloud-native containerized Kubernetes applications, Turbonomic provides invaluable insights about the resource usage of containers without requiring any extra tools. Turbonomic can also simulate a lift-and-shift movement from on-prem to public clouds for accurate estimates of the migration costs.

Principles of operation

In general, the performance of a system depends on the availability and utilization of resources. Figure 1 depicts the non-linear graph of delay, the inverse of performance, against the utilization of resources. Very low utilization implies inefficient use of resources, which results in maximum performance, while higher utilization is associated with unacceptable delay resulting in poor performance. Ideally, we want to remain at a desired state where acceptable performance is obtained at reasonably high resource utilization.

Figure 1. Graph of delay vs. utilization, highlighting the desired state

Turbonomic attempts to keep a system in the desired state using a preventative strategy rather than a reactive one. The underlying principle of Turbonomic is not based on traditional computer science but instead on the supply chain theory of economics.

Using information from the underlying infrastructure and APM tools (or in their absence from manually entered entries), Turbonomic constructs a supply chain that traverses all the way from applications to the resource providers. The supply chain is displayed in the Turbonomic GUI.

Figure 2. A portion of a Turbonomic directed graph for several cloud-native containerized and traditional applications that are deployed in on-prem virtual environments and public cloud infrastructures

The supply chain is a directed labeled graph. Figure 2 depicts a portion of the Turbonomic directed graph for several cloud-native containerized and traditional applications that are deployed in on-prem virtual environments and public cloud infrastructures. Each node in the graph has a label representing a Turbonomic entity. A node participates in a “buy” or “sell” relationship of Turbonomic commodities with other directly connected nodes. The commodities are typically resources like CPU, memory, and storage, although they can be of other varieties too.

The direction of the edges between nodes determines the buyer-seller relationship—the edges connecting nodes point from buyer to seller. For example, in Figure 2, the Container Pod node buys CPU and memory resources from the Virtual Machine node and sells the same resources to the Container node. Again, the Application Component node buys the CPU and memory resources from the Container node. There will always be a few nodes at the border of the directed graph which act either as a buyer or a seller, but not both. For example, in Figure 2, the Data Center node is purely a seller, and the Business Application node is just a buyer.

In the Turbonomic supply chain, some nodes in the directed graph can be considered pass-through nodes that do not participate in traditional resource buying or selling but are needed to understand the configuration context to execute the Turbonomic resource allocation algorithm. The Container Spec node in Figure 2 is a pass-through node.

In most cases, the edges between two nodes in the supply chain graph are unidirectional: nodes having clear buyer and seller roles. An edge can also be a bidirectional one, introducing a cycle in the graph in a few isolated instances. A bidirectional edge between a pair of nodes implies that each node both buys and sells commodities to its direct connection, such as storage and hosts in a hyper-converged cluster such as Nutanix or VMware vSAN (see the bidirectional arrow between the Storage and Host nodes of Figure 2).

Nodes in the graph contain other semantic information like the number and health of entities. To obtain a more focused view, SREs or developers can scope the supply chain to individual applications or entities in general.

The Turbonomic GUI (the supply chain graph in Figure 2) offers a single pane of glass view of applications, their components, and the resources down to the infrastructure level. This labeled graph can help SREs or developers to understand the nature of an application, its overall health, performance, and resource utilization at various levels.

In on-prem virtualized environments (non-cloud infrastructures), using the patented Turbonomic pricing and resource allocation method, Turbonomic sets the initial internal prices on the resources and provides a budget to the relevant nodes based on the demand for their resources. The prices are not fixed but follow the free-market economics principle. The cost of a resource goes up as demand increases, reducing the supply.

In the buyer and seller relationship interactions, the nodes in the Turbonomic directed graph follow the simple market principle: buy resources at a low price and sell them at a profit. For example, in Figure 2, if the memory supply runs low in a Virtual Machine (VM), a Container Pod as a buyer encounters a higher price for the memory resource. To increase its profits, the seller VM might want to scale itself up by buying more memory from a Host with sufficient available memory that can be purchased at a low price. If available, the memory buyer Container Pod will also shop for cheaper memories and might want to buy it from another VM with sufficient memory to sell at a lower price.

The Turbonomic decision engine’s market-driven algorithm continuously traverses the directed graph from bottom to top to keep the system in the desired state. During the graph traversal, Turbonomic issues recommendations holistically to assure the performance of all the deployed applications by providing them with the required resources while always maintaining an acceptable resource utilization level. For the example mentioned above, after considering the effect of its entire set of recommendations on the system, Turbonomic might decide to recommend either a memory upsize on the host VM or the pod movement to another VM.

Turbonomic environments

Turbonomic can provide tremendous value as an application resource management (ARM) platform in many commonly used environments:

• On-prem environments
• Cloud environments
• Kubernetes in on-prem and cloud environments
• Bare metal and hybrid environments

Turbonomic in on-prem environments

Turbonomic initially supported traditional virtualized on-prem infrastructures. In on-prem environments, the Turbonomic market-based economic algorithm works nicely. The price of each commodity grows exponentially as its utilization increases. Buyers shop for commodities and gravitate towards sellers offering them at the lowest price. The higher cost of a commodity can drive potential new buyers away or even an existing buyer to another seller where the commodity’s price is lower because of relatively low utilization. This movement happens as a Turbonomic redistribution action, which we talk about later in this article. Turbonomic might initiate actions to resize, reconfigure, place, provision, and deprovision resources to move towards the desired state and assure performance efficiently by reducing resource wastage.

Turbonomic in Cloud environments

Turbonomic has extended its domain of operations to cloud providers (presently AWS, Azure, and GCP) using the same buyer and seller relationship between entities of a supply chain in its directed graph. There is a subtle difference in how Turbonomic operates in the cloud compared to on-prem systems. The cloud vendor sets the price of a commodity and not Turbonomic. The concept of utilization-based pricing is not applicable in the cloud domain.

In the cloud, pricing decisions can be complex. Too many variables can determine the price of a particular group of resources, such as CPU and memory in a cloud VM. On-demand VMs usually cost much more than reserved instances (RIs). The price differences in VMs that are located in different zones and various storage categories make the situation even more complex for specific cloud vendors.

Turbonomic will generate automatable actions for using the proper infrastructure in the cloud to provide the appropriate resources to all the applications in a most economical way. Over-provisioning in the public cloud, especially for on-demand resources, can cost significant money. Turbonomic actions, when followed, will eliminate the over-provisioning costs by deleting unused or underused resources after consolidating workloads, if necessary. Turbonomic might recommend actions for moving workloads from an on-demand VM to an available RI VM of higher capacity. Turbonomic might also suggest purchasing new RIs for cost optimization. Turbonomic does consider the up-time of the on-demand VMs before making recommendations to use the existing RIs or purchase new ones.

Besides assuring application performance, Turbonomic provides actual costs incurred based on the resources used and their price set by the cloud provider. Turbonomic recommendations provide estimates of savings and investments, when applicable, in real currency. All RI-related Turbonomic recommendations contain estimated cost savings and include estimated investment costs if they have RI purchase recommendations.

Turbonomic in Kubernetes in on-prem and cloud environments

Microservices and other enterprise-class, cloud-native containerized applications mostly run under Kubernetes. Kubernetes workloads can be deployed in on-prem environments or cloud environments.

The scalability, self-healing, and other sought-after characteristics of Kubernetes demand a higher level of sophistication from an ARM platform. Using the market-driven principles mentioned earlier, Turbonomic works in near real-time in both the categories of the environments dealing with the ephemeral entities and provides the needed agility and responsiveness.

Let’s consider these features of Kubernetes and how Turbonomic works with them:

• Requests and limits
• CPU throttling
• Noisy neighbor

Requests and limits

Containers in multi-tenant Kubernetes environments typically have requests and limits values specified in their deployment specifications for CPU and memory. In Kubernetes, the usual unit for CPU is millicore and for memory is mebibyte (MiB). Figure 3 shows the values for the Payment container of one of the Kubernetes business applications in Figure 2.

Figure 3. Resource requests and limits for the Payment container

Before placing a pod in a node, the Kubernetes scheduler ensures that the containers in the pod will always receive at least the amount of CPU and memory values that are specified in the requests attributes for the containers. The following relationship must remain satisfied for both CPU and memory during pod scheduling and at any time.

(Sum of the ‘requests’ values for all the containers) < (The capacity of the host node)

The limits values indicate how much CPU and memory a container can acquire. If no limit is specified for CPU or memory for a container, while executing, the container can acquire an unlimited amount and potentially monopolize the host VM's entire CPU or memory capacities. For this reason, developers almost always set the limit values before deploying the containers. If the requests attribute for a resource is missing in the specification, Kubernetes implicitly sets its value as specified in limits during deployment.

The Kubernetes scheduler does not consider limits while scheduling pods. In a sense, this almost always results in pod over-scheduling in a node. That is, the total amount of CPU or memory requested by all the containers in a node at a time might exceed the total CPU or memory resources available in the node. CPU is a compressible resource, while memory is an incompressible one. Over-scheduling of pods might squeeze the available CPU among competing containers, while the same for memory might create out of memory (OOM) exceptions in containers and therefore the containing pods. If a container attempts to acquire memory beyond the specified limit, it will encounter an OOM exception. If a container needs more memory than specified in the request, it will get it, provided the amount requested is available in the node; otherwise, it might get evicted with an OOM exception. Kubernetes might restart the affected pods, but the OOM cycle might continue if the memory demands of containers remain the same.

The Kubernetes scheduler attempts to use the available nodes and the associated resources in a practical fashion. If Kubernetes considered the values of the limits while scheduling pods, many worker nodes would have been needed to host all the pods in most environments, and perhaps a significant number of these worker nodes would have remained underutilized for most of the time.

Turbonomic always considers the CPU performance and pricing of the infrastructures under consideration. It uses the underlying chipset configuration and the SPECint benchmarking for on-prem environments. For cloud environments, Turbonomic uses similar data enhanced with the addition of the cloud vendor benchmarking for the instance families. Turbonomic uses the CPU performance information to map the millicore CPU unit of Kubernetes to the GHz unit of the physical CPUs of the infrastructures.

Kubernetes expects developers to set appropriate values for the requests and limits. Two issues can arise when the values for the requests and limits are not properly set: CPU throttling and noisy neighbor. CPU throttling (described in detail below) happens all the time in enterprise Kubernetes environments; it is a price one has to pay for multi-tenancy. Using too low values for requests increases the probability of creating resource congestion by having too many pods in worker nodes -- the noisy neighbor issue (see below). High values for the requests may waste resources, with the Kubernetes scheduler creating underutilized worker nodes. Too low values for limits might not provide enough resources for containers to execute, affecting the SLOs of the corresponding applications. Too high values for limits can create OOM exceptions in some containers or monopolization of CPUs by some others.

Kubernetes uses the static values of requests and limits. It cannot dynamically tune the values of these two attributes based on the resource usage of the containers. The appropriate setting of requests and limits might not be an easy task. The activity might need a few iterations of load testing in real-life environments or simulations.

CPU Throttling

The CPU limits attribute can create interesting CPU congestion situations in Kubernetes environments that are often challenging to diagnose. Conventional monitoring tools might show low CPU utilization in a container that might prompt SREs to lower the CPU limits value of the container for achieving overall better performance for other co-located containers in the same node. However, SREs or developers might observe that lowering the CPU limits significantly slows down the container. It is not uncommon to see that while a node has a low CPU utilization some otherwise healthy containers hosted in the node are slow in executing computations, indirectly indicating possible CPU starvation.

This somewhat counterintuitive behavior is because of how Kubernetes enforces the CPU limits by using the time quota of the completely fair scheduler (CFS) of Linux and not the available CPU in the node. By default, the value of the time quota is 100 milliseconds (ms). Kubernetes uses the CFS to limit the CPU usage for a container by taking the CPU away in each time quota slice after a period of milliseconds as calculated by the following formula:

((CPU limit value of the container specification) / (The CPU capacity of the host machine)) * (Time quota of the Linux CFS Scheduler)

Losing CPU in time quota slices because of limit enforcement by Kubernetes is commonly known as CPU throttling.

Figure 4 shows a slightly simplistic case of CPU throttling for a single-threaded execution of a request to the Payment container (see Figure 3). In this example, the host is a single-core (1000 millicore) machine and the request needs 100 ms of CPU time to complete.

Figure 4. Simple case of CPU throttling for the Payment container

Using the previous formula, for every time quota slice, the Payment container will get a CPU time slice of ((200 / 1000) * 100) milliseconds, which is 20 ms. The remaining 80 ms of CPU time will not be given to the container even if there are no other users on the machine. Therefore, a 100 ms activity will take four full and one partial time quota slice to complete, as shown in Figure 4.

In this specific case, the total time needed to complete the request is (4 x 100 + 20), which is 420 ms, which is a serious performance degradation of (100/420) x 100, which is 23.81%. If the Payment container had either not specified the CPU limits or had set the limits to the full CPU of the host (1000 millicore), the task could have been completed in just one time quota slice of 100 ms. Though the fundamental principles remain the same, multi-threaded computation makes the situation somewhat more complex.

Turbonomic quantifies CPU throttling as:

T = ((n / (n + 1)) * 100) %

In this formula, T is the CPU throttling percentage, and n (n >= 0) is the number of time quota slices when the CPU is taken away during computation.

For the Payment container in our example, the value of CPU throttling is (4 / (4 + 1)) * 100, which is 80%. From this formula, we can infer that to keep the CPU throttling percentage T low, we should have a low value of n. For a very high value of n, the container will get only an insignificant amount of CPU, resulting in a very high value of T.

Noisy neighbor

In Kubernetes environments, containers typically share nodes. Often in a node, one or more containers acquire so much CPU that other containers can experience CPU starvation. A group of co-located well-behaved containers can all reach the peaks of their CPU demands at about the same time. This can cause CPU starvation among themselves. For memory, similar situations might result in OOM exceptions and consequent restarts of pods, as mentioned previously. This issue arising from co-located consumers is commonly known as noisy neighbor.

Because the Kubernetes scheduler does not consider the limits value while placing pods, a noisy neighbor can unexpectedly slow down and adversely impact other applications sharing the same node. The noisy neighbor problem might not be easy to detect in a real-life environment in real-time. To pinpoint the offending containers, one might have to go through all the containers’ CPU and memory consumptions in a node.

Turbonomic in bare metal and hybrid environments

For Kubernetes workloads, Turbonomic actions that deal with the redistribution of pods and the scaling of containers and namespaces (see Table 3 of the Scaling actions of Turbonomic section) work in unison in bare metal infrastructures. Turbonomic does not provide much value for traditional applications deployed in bare metal environments.

In a multi-cloud or, more generally, in a hybrid environment consisting of bare metal servers, on-prem infrastructure, and one or more supported cloud infrastructures, Turbonomic will work in the following fashion:

• With proper APM tools, the supply chain directed graph will be comprehensive and can span across multiple infrastructures
• The redistribution and scaling recommendations will be scoped to individual infrastructures.

Turbonomic does not currently support Functions as a Service (FaaS) environments.

Turbonomic actions

Turbonomic recommends actions on the application hosting environments. The actions are all explainable, and most of them are automatable. SREs or developers can see all the actions recommended in an environment and drill down to the details of the individual actions in the Turbonomic GUI (directed graph). The details of any particular action in the GUI provide clear explanations behind the recommendations and the pre-effects and the predicted post-effects of the action on the concerned buyer and seller entities in resource consumption.

The effect of most of the Turbonomic recommended actions can be of two categories: performance and efficiency:

• The performance actions adjust the environment so that the buyer entities get the resource they need from sellers in a proactive manner before they exhaust the available resources and cause application outages or severe performance degradations.
• The efficiency actions address the under-utilization and over-provisioning of resources in seller entities. For economical use of resources, this category of actions increases the resource utilization of the sellers by consolidating buyers in an attempt to suspend one or more sellers. Turbonomic will always consider application performance before recommending any actions for increasing efficiency.

Some actions might result in both performance and efficiency improvements. Prevention is another category of actions, and they too attempt to keep the system in the desired state. In a sense, we can consider the prevention actions belonging to performance and efficiency categories. Compliance actions are the final category of actions, and they correct the misconfiguration of entities associated with user-entered placement or access policies.

From the operational viewpoint, the actions recommended by Turbonomic can be broadly classified into two categories:

• Redistribution of buyer entities. To avoid resource congestion, Turbonomic might recommend moving VMs from one physical host to another or moving container pods from one node to another. Turbonomic might also recommend buyer redistribution for their consolidation. The redistribution in distributed computing environments is typically for performance, efficiency, or both.
• Scaling of resources. For better performance, Turbonomic might recommend the acquisition of more seller entities (provision new VMs or new physical hosts) as upward scaling actions. For efficiency, Turbonomic might also suggest removing sellers (suspend VMs or physical hosts, release acquired storage, and so on) as downward scaling actions. See the next section for more details about scaling actions.

The Turbonomic resource allocation algorithm does not consider only individual buyer and seller nodes while formulating actions. It traverses the entire directed graph and provides a comprehensive list of actions covering all the buyer and seller nodes in the directed graph. After all, the task of Turbonomic is to bring and keep the entire system in the desired state without focusing only on individual entities.

Turbonomic collects granular resource utilization data slices from various targets (built-in platform tools or APM utilities when they’re available) every 10 minutes and stores the data in an internal database for a certain amount of time. Turbonomic uses the stored data for analysis. Turbonomic must quantify and express the present and projected utilization of all the resources to recommend comprehensive groups of actions and provide clear explanations. In almost all cases (with the exceptions of a few specific instances), Turbonomic uses percentile calculations and not average values of resource usage. The use of percentiles eliminates sizing to outliers while formulating scaling actions, such as infrequent CPU peaks encountered during application starts will probably not get considered in scaling actions using percentile mathematics. To identify sound recommendations, Turbonomic bases its decisions on an observation period of resource utilization.

Turbonomic sets the default value for the observation period and the percentile values for resources like CPU, memory, and so on. These default values are platform-dependent and will provide the most appropriate recommended actions for most environments. For example, 95 is the default percentile value in the on-prem platforms, and the default observation period is 30 days. However, one can adjust these values in a few discrete steps and within certain limits to suit the needed analysis sensitivity for specific environments. An upper percentile value will assure performances for ultra-critical applications or applications that must be able to nicely handle infrequent and unpredictable resource spikes at the cost of overall inefficient resource usage. In contrast, a lower percentile value might be perfectly suitable for development environments for more economical and conservative use of resources. A higher value of the observation period will result in more conservative scaling actions, while a lower observation period may result in frequent scaling actions.

Scaling actions of Turbonomic

Turbonomic often recommends scaling actions on relevant resource entities for on-prem and cloud infrastructures. A scaling action on an entity can be horizontal, which implies changes in the number of instances of the entities, or vertical, which suggests changes in the size of an entity. Scaling direction can be up, which is accomplished by increasing the number of instances or entities' size, or down, which is associated with reducing the number or size of the entities under consideration.

After a seller resource category horizontally scales up, Turbonomic will move relevant buyer entities to the new instance. Turbonomic might also recommend suspension or redistribution of existing buyer entities to other seller entities before a seller entity type gets scaled down horizontally. Turbonomic recommendations always consider user-specified placement constraints for VMs and pods for all relevant scaling actions.

A significant portion of the scaling actions can be performed automatically by Turbonomic, while the rest need some sort of manual intervention. The nature and extent of manual intervention are dependent on the underlying infrastructure provider.

Table 1 summarizes the scaling actions of Turbonomic in on-prem environments at a level of abstraction. Turbonomic recommends horizontal scaling of VMs to handle Kubernetes workloads.

Table 1. Scaling actions of Turbonomic in on-prem environments

Table 2 depicts the scaling actions of Turbonomic in cloud infrastructure again at a level of abstraction. Like on-prem environments, the horizontal scaling of VM actions in cloud infrastructures is associated with Kubernetes workloads.

Vertical scaling on VMs in the cloud works differently from on-prem infrastructures. In the cloud, when applicable, the Turbonomic vertical scaling action logically changes an existing VM with an appropriate VM of another instance type or tier. The reserved instance (RI) vertical scaling is an essential feature of Turbonomic in public cloud environments. In the case of RI vertical scaling, even for scale-down actions, for cost optimization, Turbonomic might recommend substituting an on-demand VM with a larger RI.

The storage scaling in the public cloud is mainly of vertical and deletion (horizontal scale down) varieties. The actions and their mode depend heavily on the features provided by the public cloud provider. Scaling actions might need redistributions of existing workloads. For non-Kubernetes workloads, the redistribution mechanism can be cloud provider dependent.

Table 2. Scaling actions of Turbonomic in cloud environments

Table 3 focuses on the scaling of Kubernetes. Kubernetes workloads can execute on both on-prem and cloud infrastructures. In a sense, one can append Table 3 to Table 1 and Table 2 to have a complete picture of scaling when Kubernetes workloads are hosted in on-prem and cloud infrastructures.

Table 3. Scaling actions of Turbonomic in Kubernetes

Note: With the exceptions of performance metrics collecting pods, and the DemonSet pods, Turbonomic does not provide any recommendation on the horizontal scaling of pods. Turbonomic uses the data from the metrics collecting pods to check the SLOs of applications regarding application response time and transaction throughput. Turbonomic recommends provisioning or suspending actions on DemonSet pods as part of horizontal worker node scaling actions.

The horizontal pod autoscaling feature of Kubernetes performs the horizontal scaling of pods (and hence containers) as mentioned in the first two rows of Table 3.

Turbonomic plays a unique and interesting role in the vertical scaling of containers. To resize containers, Turbonomic adjusts the value of the limits of the container specification. Refer to Figure 3, where the bidirectional arrow indicates that Turbonomic can change limits in either direction. Turbonomic recommends to increases the CPU limit to reduce CPU throttling (Figure 4). A recommendation to increase the memory limit can potentially provide more memory to the container and thereby minimize or eliminate the chance of receiving an OOM exception for any container that does not leak memory. Turbonomic also uses historical information about past OOM exceptions, if any, from a container to upsize its memory limit. Turbonomic can also recommend decreasing the limits to provide better resource utilization by potentially increasing the chance of placing more pods in the underlying namespace. As part of vertical scaling for containers, Turbonomic might recommend reducing the values of the requests for CPU or memory. Lowering requests values can help the Kubernetes scheduler place more pods in the worker nodes, thereby increasing their utilization.

Turbonomic might recommend increasing CPU, memory, or both, quota values for namespaces as another variation of vertical scaling. This action might be necessary to accommodate pods in a namespace before the relevant containers scale up vertically.

Turbonomic does not recommend any increase in the values of the requests attribute. It also does not recommend decreasing the values of the namespace quotas. If deemed necessary, these adjustments have to be done by users outside Turbonomic.

Unlike Kubernetes, Turbonomic can dynamically adjust the resource limits and requests (scale down only). This can be of great value to container developers. Developers can provide initial requests and limits for containers and decide to tune the static values later based on the values recommended by Turbonomic in QA or production environments.

All the replicas of a container must be identical, which means for any vertical resizing, all the replicas of the container will be identically resized. If there is not enough cluster capacity to accommodate upsized containers, Turbonomic will recommend an action to provision new nodes before allowing the user to accept the container scaling action.

Container resizing needs a restart. Turbonomic merges individual vertical resizing actions of containers into one consolidated action to minimize the number of restarts. For example, Turbonomic will merge a CPU limit increase action and a memory limit increase action on the same container to one action. The pod restarts of the vertically scaled containers do not result in any application downtime. Turbonomic uses the Kubernetes rolling update feature to restart the vertically sized containers.

Redistribution actions of Turbonomic

The redistribution actions recommended by Turbonomic are for avoiding resource congestion and workload consolidation. Turbonomic can redistribute VMs in on-prem infrastructures to different hosts and pods to other nodes in a Kubernetes environment. Consolidation of VMs and pods might be needed to suspend hosts and VMs on-prem. Consolidation of pods might also be necessary to suspend VMs in the cloud.

Turbonomic actions can spread out pods more evenly across worker nodes than initially placed by the Kubernetes scheduler to avoid resource congestion. However, Turbonomic is not a load balancer, and it might not distribute pods evenly across all the worker nodes.

Turbonomic recommended actions about decreasing the CPU limit (scaling) and relocating (redistribution) relevant pods can minimize or eliminate the noisy neighbor issue in Kubernetes deployments.

For relocating on-prem VMs in a non-disruptive way, Turbonomic depends on the specific mechanisms of the infrastructure providers. Pod relocation can be viewed as a logical concept. To relocate a pod, a clone of the source pod is first created in the destination environment, and later, the original pod gets deleted.

Turbonomic recommended redistribution of entities always abides by user-specified placement constraints for VMs. For Kubernetes, Turbonomic redistribution actions honor existing node taints and pod tolerations constraints.

Holistic nature of Turbonomic actions

Turbonomic considers the total and complete effect of all the related actions on all the applications holistically across the environment before recommending any action. Turbonomic never issues any recommendation in isolation. Its algorithm carefully considers the combined effect of all the actions on the performance of all the targeted applications while considering the efficient use of all the relevant resources. Figure 5 shows a merged action comprising two separate vertical scaling actions on the Container Spec Payment, one dealing with CPU throttling and the other with memory congestion of the Payment container.

Figure 5. Merged actions for vertical scaling of a container spec

The Turbonomic merged scaling action recommends vertically scaling up the limits of the Payment container specification from 200 millicores (mCores) to 500 millicores for CPU and from 100 mebibyte (MiB) to 128 MiB for memory, as shown in Figure 6. After the actions are taken, the CPU throttling and memory congestion in payment will come down.

As highlighted in Figure 6, the effect on the container spec as estimated by Turbonomic is not because of the merged action alone. The estimate is the improvement from all the actions pending in the environment. Turbonomic always considers the total effect considering all the actions at a given time; it never arrives at an estimate considering only one action in isolation, which may not be accurate in most instances.

Figure 6. Impact of all recommended Turbonomic actions for the container spec payment

Notes on Figures 5 and 6:

• Containers are ephemeral. The Container Spec entity is persisted, and it contains the historical utilization data for all the replicas of a container. Turbonomic uses the stored data. As shown in Figure 6, Turbonomic displays the vertical scaling recommendation action for a container and the effect of all the recommended actions in the context of the associated Container Spec entity.
• Turbonomic recommended actions for vertical scaling of a container (Container Spec) are displayed in the scope of its workload controller (which is a Kubernetes controller), as shown in Figure 5. An application is a workload in the Kubernetes world. An application can consist of more than one container, and hence a Turbonomic workload controller might also contain vertical scaling actions related to several containers.

This holistic approach distinguishes Turbonomic from cloud providers and other vendors in the resource optimization space, where individual actions like the vertical scaling of containers are considered in isolation. The same observation applies to Kubernetes and its various vendors regarding the Cluster Autoscaler and the Vertical Pod Autoscaler. Both these scaling mechanisms are trigger-based and are unidirectional in nature. None of these two mechanisms consider their effects on the entire Kubernetes deployment. As a result, they might fail to satisfactorily address the desired performance and resource utilization issues.

Interestingly, in the Kubernetes platform, Turbonomic scaling recommendations theoretically work in n (n<=9) different dimensions in unison with Kubernetes. Recall from Tables 1, 2, and 3 that the dimensions can be:

1. Horizontal scaling of on-prem hosts or RI scaling for cloud VMs
2. Horizontal scaling of VMs (worker nodes in Kubernetes)
3. Vertical scaling of VMs
4. Horizontal scaling of pods (done by Kubernetes)
5. Vertical scaling of containers
6. Vertical scaling down of guaranteed resource availability of containers
7. Vertical scaling up of namespace quota
8. Horizontal scaling of storage
9. Vertical scaling of storage

The first list item is infrastructure dependent, and Kubernetes performs the fourth. In Kubernetes environments, one can view Turbonomic as a higher level orchestrator of containers working in conjunction with the base Kubernetes orchestration.

Using the same holistic category of actions, Turbonomic can simulate various scenarios by executing pre-defined plans, such as optimizing cloud infrastructures, optimizing on-prem infrastructures, optimizing the number of worker nodes for a Kubernetes cluster, migrating on-prem VMs to a cloud, migrating VMs from cloud provider A to another cloud provider B, on-prem hardware change, and so on. SREs can create custom plans too.

Turbonomic always provides a cost impact of the plan's actions when a cloud is involved. The cost estimates related to the plan’s action do not include network-related costs, which users can add manually for more accurate estimates. Note that cloud migration planning analyses are point-in-time statements. Cloud providers frequently change the pricing and specifications of compute instances and storage. The cloud-related migration analysis can potentially become stale in a relatively short time frame.

Conclusion

Turbonomic continuously monitors environments, collects metrics from various abstraction layers, and provides actionable, preventative recommendations to mitigate performance risks and to assure that their applications can meet their SLOs. Besides being a useful ARM utility for enterprises, Turbonomic provides excellent value in some other related situations too:

• Turbonomic can provide guidance to reduce waste in public cloud environments. This can result in significant cost savings.
• Turbonomic can be used as a debugging and performance tuning utility in Kubernetes environments. By following the Turbonomic recommendations, developers can easily adjust container resource specifications to mitigate CPU throttling and noisy neighbor issues.
• Turbonomic can provide cost estimates in real currencies for various migration scenarios involving clouds. This can be of great value to the stakeholders before deciding and budgeting on a migration project.

Business-critical applications often slow down in real-life environments, which typically creates traditional or virtual war rooms. In a war room, manual interpretation of performance data from disparate sources, unproductive heated debates between technical experts dealing with applications, infrastructures, and so on, often lead to erroneous decisions made in haste. An adequately deployed Turbonomic utility can eliminate the war rooms and the associated chaos in most instances by providing clear reasons for slowdowns along with necessary remedial recommendations. In a true sense, Turbonomic can be the very best friend and assistant of SREs.

Perhaps the most significant value of Turbonomic is realized in environments where most of the common redistribution and scaling actions are automated. Turbonomic automation can be realized straight out of the box with no script management or maintenance from the end-user. Although new users might be somewhat hesitant towards comprehensive action automation, the task of automating actions can be realized in a few phases as the comprehension and confidence of users in the explanations behind actions increases. After a good portion of the actions gets automated, Turbonomic will proactively take care of most of the performance issues of all the reasonably well-tested enterprise quality applications in complex environments with minimal or no operator intervention.

Turbonomic and its algorithm will continue to evolve. It will not be unreasonable to expect future enhancements from Turbonomic in the fast-growing area of hybrid platforms. Turbonomic might also collaborate and integrate with other products. For example, the recent integration with Flexera has added the capability of optimizing software license costs in cloud environments (which we plan to discuss in a future article).

Acknowledgments

Debasish acknowledges the valuable discussions with Laura Scott, VP, regarding the use cases of Turbonomic. Laura and Peter Guglielmino, DE, reviewed the initial draft and provided suggestions for improvements. Debasish received his serious Turbonomic education from David Fiore, Divya Singal, and Rob Cooper. Special thanks from Debasish to Laura, Peter, and Aboud Ghazi for all the discussions, guidance, encouragement, and support.

Justus acknowledges the mentorship of Anson McCook and Jimmy Herbert to be able to differentiate the Turbonomic analysis in the marketplace. Additionally, Eva Tuczai was integral in enhancing Justus’ Kubernetes foundation, and Bobby Coppersmith helped Justus understand where the native platform falls short with analysis.

The authors also thank Michelle Corbin and John Meegan for their work in publishing this article. Michelle patiently went through multiple drafts to improve the quality of the content in a short time frame. John shepherded the entire publication process over several meetings and provided many helpful suggestions.