Understanding the data collection and action execution mechanisms of Turbonomic

IBM Turbonomic is an Application Resource Management (ARM) platform for on-prem, cloud, and hybrid environments. As an ARM, Turbonomic automatically ensures the performance of a system by providing the optimal amount of resources to applications in near real time.Interestingly, Turbonomic’s resource optimization principle is based on fundamental principles of economics, not traditional computer science.

Informally speaking, Turbonomic collects performance and resource utilization metrics about applications and the hosting environments from various sources, such as:

• Application Performance Management (APM) utilities, such as Instana, Dynatrace, and so on
• Kubernetes
• On-prem hypervisors or cloud providers

The data collected from various applications and environments are analyzed by a patented, unique Optimization Engine of Turbonomic, which attempts to bring the aggregated performance of the entire environment to an optimal state, proactively assuring the performance of the hosted applications by providing the optimal amount of resources to them, eliminating both the cases of under-provisioning, which negatively impacts application performance, and over-provisioning, which results in resource wastage.

The Optimization Engine works on an “economy,” which can be viewed as a dynamic, directed, and merged graph of traders buying and selling different commodities. Figure 1 shows a typical Turbonomic economy. The nodes in the graph correspond to traders who buy and sell commodities like vCPU, vMem, vStorage, CPU, Memory, and so on. The edges in the graph depict the buyer and seller relationship. The arrows point from buyers to sellers of commodities.

Figure 1. A Turbonomic directed graph

In Figure 1, the Container Pod traders buy commodities from the Virtual Machine traders. Turbonomic displays the directed graph of an economy in a ‘merged’ form by collapsing the same type of traders in a single node. The integer inside the nodes indicates the cardinality of that type of trader. In Figure 1, there are 328 Virtual Machine traders. During the lifetime of an environment, the traders, their numbers, and the relationships between them might change, implying the potentially dynamic nature of the system and, hence, that of the economy graph.

The graph in Figure 1 is called a “supply chain” in Turbonomic literature.

The Optimization Engine attempts to bring an environment to its “desired state.” A system's desired state is defined to be an integer in a closed interval [x, y], where x < y < 100. The desired state value represents the percentage utilization of resources.

Within the desired range of resource utilization, a system's aggregated performance remains optimal. Resource utilization below x, the lower limit of the desired state, wastes the available resources in the environment. On the other hand, pushing resource utilization beyond y, the upper limit of the desired state will hurt the aggregated system performance because of reasons like multiple applications peaking together or a few applications acquiring more resources, creating resource scarcity for others. (See the Product Overview section of the Turbonomic documentation for the operational details of the desired state.)

The output of the Optimization Engine is a set of recommendations or actions to be executed on the environment for its reconfiguration. In abstraction, most actions can be classified into two distinct categories:

• Scaling (horizontal and vertical). The Optimization Engine might generate horizontal and vertical scale-up actions on virtual machines (VMs), physical hosts, and so on, to increase the available resources, depending on the environment and specific situations. Similarly, the Optimization Engine may generate horizontal and vertical scale-down actions to reduce resource waste.
• Redistribution of traders in the environment under consideration. The Optimization Engine might generate redistribution actions redistributing VMs between physical hosts in on-prem environments, Kubernetes pods between worker nodes, and so on, to appropriately allocate available resources.

One unique, powerful, distinguishing feature of Turbonomic that sets it apart from others is its ability to automatically execute most of the actions generated by the Optimization Engine by invoking the relevant APIs of the underlying platforms.

The raw data collected from the APM utilities, when present, and the providers of execution environments might need normalization and some other preprocessing before the Optimization Engine can use them properly.

The eternal cycle of the Turbonomic runtime

Turbonomic probes are the software components that perform the following activities in general:

• Probes collect metrics and other relevant data from APM utilities, hypervisors, and clouds so that the collected data can be normalized and pre-processed as necessary before handing the data over to the Optimization Engine.
• Probes invoke the appropriate APIs of the hypervisors, clouds, and Kubernetes system to execute the actions generated by the Optimization Engine.

The Turbonomic runtime follows an eternal cycle, as shown in Figure 2. The probes collect data. Then, after normalization and any necessary preprocessing of the collected data, the transformed data is sent to the Optimization Engine for analysis. After analyzing the present state of the entire environment, which includes all the hosted applications, the Optimization Engine might generate a set of actions. The actions are again executed through probes using the appropriate APIs of the underlying platforms.

Figure 2. Lifecycle of the Turbonomic runtime

When executed in its entirety, the set of actions generated by the Optimization Engine will bring a static system to its desired state, as mentioned earlier. However, in real life, systems have inherent dynamism associated with them – the user load on the applications changes, brand new applications might get deployed, existing applications might get upgraded, old applications might be retired, hardware changes might happen in on-prem environments, and so on. Because of its inherent dynamism, a system might gradually drift away from the desired state. In real life, not all actions will get executed in one shot, which will also keep the system away from the desired state.

Because of these reasons, Turbonomic continues to repeat the execution cycle of the Data Collection and Pre-processing, Analysis, and Action Execution phases ad infinitum. The cycle repeats every ten minutes by default. Advanced users of Turbonomic can think of fine-tuning the cycle repetition time based on the environment's size and nature. Continuing the cycle endlessly at a short interval ensures that, provided the generated actions are executed, the system will remain close to the desired state most of the time.

The probes are the critical interfaces of the Optimization Engine with the outside world. For a thorough understanding and working knowledge of Turbonomic, one should have a sound knowledge of the Turbonomic probes, their architecture and working principles.

The endless lifecycle of Turbonomic introduced here is a new abstract model of Turbonomic, which will help us understand, use, and enhance Turbonomic. This article discusses Turbonomic probes in detail at a level of abstraction without delving too much into implementation details. After reading this article, you should have a sound understanding of the Turbonomic probes, which will significantly help you use the probes properly in Turbonomic deployments.

The basic concepts of Turbonomic probes

Turbonomic probes have configuration entities called “target types.” As part of the standard Turbonomic installation, certain pre-defined Turbonomic target types, which are defined in a Custom Resource (CR) file, get deployed through a custom Kubernetes operator. Every target type is associated with a probe. The deployment of a target type automatically deploys its associated probe. Probes and, hence, their associated target types can be enabled or disabled by appropriately modifying the CR file. For details of enabling or disabling probe components, refer to the General Configuration Tasks Chapter in the Turbonomic documentation.

Turbonomic probes are container images that execute in their pods. New software releases that contain enhancements or bug fixes for specific probes can be pulled from the repositories and activated by restarting the relevant probe pods.

Users can create new targets from the available target types or modify existing targets using the Turbonomic GUIs or Rest APIs (see the Target Configuration section or the API Reference section of the Turbonomic documentation). Creating a target from a specific target type can be viewed as creating and activating instances of the probe associated with the target type.

A typical Turbonomic probe is designed to collect various data from and execute actions in various infrastructure and application hosting environments like on-prem hypervisors, clouds, and Kubernetes by invoking proper APIs of the underlying environments.

There are probes for collecting detailed application-level metrics from APM, but the APM probes, in general, do not execute any actions. There are also probes that do not collect data or execute any actions but are used for tasks like getting the generated actions from the Optimization Engine approved through IT Service Management (ITSM) software and other specialized activities.

Probes have built-in information about the traders and the types of actions they execute on them. For example, the vCenter probe provides the information that it acts on the Virtual Machine traders to scale or move them. As another example, APM probes specify that they do not execute any actions on any traders. The information regarding the specifics of action executable by probes can be considered metadata of probes available internally to the relevant Turbonomic component dealing with action executions.

A target, and hence a probe instance, for a specific environment gets created from a set of user-specifiable configuration attributes, like the addresses (endpoints for on-prem hypervisors, accounts for clouds, and so on) of the targeted environments and the necessary user credentials needed, if any, to connect to the addresses to gather relevant metrics and execute actions, when applicable. Depending on the underlying environment and the intended use of the probe, the user credential of the target must have the necessary access rights. A probe instance can essentially be a ‘read-only’ probe if the user credential of the associated target does not have the authority to manage (reconfigure all or certain aspects of) the environment for executing the Turbonomic-generated actions.

A Turbonomic deployment can have multiple target types for the same environment, each associated with a different address and possibly having different capabilities. For example, both the “AWS” and “AWS Billing” are target types of the AWS environment associated with different AWS accounts. The probe corresponding to the AWS target type will collect utilization-related metrics, while the probe configured with the AWS Billing target type will collect cost-related information for the specific account from AWS.

Each target in a Turbonomic deployment must be unique. The uniqueness of a target is based solely on the address of the environment configured in it. Other attributes, like user credentials, do not play any role in defining uniqueness. The Turbonomic runtime might throw exceptions if a user configures multiple targets having the same addresses for their environments.

Note that a target type can have n (n >= 0) targets in a Turbonomic deployment. For example, a vCenter target type can have n (n > 0) targets; for multiple vCenter targets, each must have a different vSphere server address corresponding to a vSphere cluster. A target type yet to be used to create any target trivially satisfies the condition of having zero targets.

Although most target types can have any number of targets, because of economic and other considerations, a target type can be restricted to only a specified number of targets and, hence, the same maximum number of probe instances in a Turbonomic deployment. The restriction is programmatically enforced in the Turbonomic API code for configuring targets, which can be invoked directly from the Turbonomic GUI to create or modify targets. Presently, only the ServiceNow target type for the ServiceNow ITSM software has such a restriction. The ServiceNow target type can have only a single target.

In a sense, a probe software component defines the “probe type” Pi associated with a target type Ti (i >= 1), while its instantiations are runtime probe instances pij are linked to specifically configured targets tik. Formally speaking, the relationship between probes and target types (Pi <--> Ti, i >= 1) is bijective; that is, for each probe Pi, there exists a unique target type Ti, and similarly, for each target type Tj, there exists a unique probe Pj, where i, j >= 0. Figure 3 schematically depicts the relationship between probes and target types.

Figure 3. Bijective relationship between probes and target types

For most probes, there exists a one-to-one bidirectional mapping between probe instances and targets. However, there are a few probes for which an instance can have two modes of operations: default and incremental. (These modes are explained later in the “Data Collection and Preprocessing” section). Potentially, these two modes can be executed in parallel. For these dual-mode probes, a single probe instance gets cloned into two instances, each associated with a unique mode of operation, with both instances executing in parallel. For this reason, the relationship between probe instances pij and targets tik (pij --> tik where i, j, k >= 1 and j <= 2k) should be considered surjective, as shown schematically in Figure 4. There is at least one probe instance for every target, and all probe instances have associated targets. Sometimes, two probe instances can be linked to the same target.

Figure 4. Surjective relationship between probe instances and targets

Turbonomic probe instances are typically stateless. Their operations are not dependent on any data from previous executions, if any.

A Turbonomic probe is encapsulated in a container. A probe instance executes as a thread in the container allocated for the corresponding probe.

In Turbonomic literature, the terms “probe” and “target” are often used interchangeably. In the context of probes, target implicitly implies that probes target specific environments, such as platforms, utilities, middleware, and so on. Usually, this does not cause any confusion. Strictly speaking, though, a probe is a containerized software component that uses a corresponding target at instantiation to connect to the intended address and become functional at runtime. In this article, we might use the term probe for “probe instances” when there is no chance of confusion.

See the Target Configuration section of the Turbonomic docs for details on the available target types and target configurations.

With the sole exception of the Turbonomic probe for the Kubernetes environment, none of the Turbonomic probes are agents of any kind. Turbonomic probes do not have any footprint outside Turbonomic deployments, and they work by invoking appropriate APIs. You do not need to upgrade probes if the underlying application hosting environment gets upgraded to newer versions of platform software until and unless the upgrade alters or removes any underlying API definitions that are used by the probes.

For optimizing Kubernetes platforms, Turbonomic uses a unique dedicated pod, the KubeTurbo pod, which is deployed in the user’s Kubernetes cluster to collect data and execute actions in it. The Turbonomic runtime communicates with KubeTurbo using a secure WebSocket connection. Theoretically, there can be potential mismatches between the Kubernetes versions of Turbonomic’s KubeTurbo component and the underlying Kubernetes platform to be optimized. The mismatches in Kubernetes versions can arise because of the Kubernetes version upgrade on the user Kubernetes cluster. The backward compatibility of the Kubernetes APIs should not create any issue for Kubernetes version mismatches; however, IBM intends to incorporate a strict compatibility check between the Kubernetes versions of KubeTurbo and that of the user’s Kubernetes cluster.

In a SaaS Turbonomic instance that optimizes on-prem platforms, probes invoke the relevant APIs of the on-prem platform through the SaaS Client using the secure connector. For details on the SaaS client, see the SaaS Management section of the Turbonomic documentation.

Data collection and preprocessing collected data in Turbonomic

To better understand Turbonomic probes, let’s review the details of the architecture of Turbonomic for collecting metrics and other data from APM utilities, underlying platforms, and so on by using probes. Then, let’s review the preprocessing of the data that Turbonomic does before sending it over to the Optimization Engine for analysis (see Figure 1 above).

Data collection interval

At predetermined intervals, probes call the relevant APIs of the endpoints to collect the required infrastructure information and utilization metrics. The time interval of API invocations for data collection by probes can be explicitly specified in an XML configuration file for each probe. In the absence of being explicitly specified, the data collection interval defaults to ten minutes, which is the case for most probes. However, certain probes might have higher default collection intervals. Relatively higher time intervals can sometimes be necessary to prevent flooding the targeted environment with too many API invocations or from being blocked by the endpoint provider. For example, the storage probe for Microsoft Azure has the default collection interval of four hours because Azure will block frequent calls to the Azure storage endpoint.

Because the time interval is associated with a probe, all the probe runtime instances associated with a probe will have the same frequency of data collection. In almost all cases, the out-of-the-box data collection time interval, either default or explicitly specified, will suffice.

Depending on the provider, the frequency of data collection, and the specific user account that is used, API invocations by probes might get blocked in the cloud domain. In such cases, you might have to adjust and increase the time interval of data collection by probes. A higher value of the collection interval will result in fewer API invocations, which might result in cost savings because cloud providers typically charge for individual API invocations. Before increasing the data collection time interval for pure cost savings, you should carefully consider the fact that Turbonomic executes a near real-time optimization based on the collected metrics from probes, and too much delay in the metric data collection can introduce staleness, which in turn might result in subpar optimization results in specific instances.

Collected data

All Turbonomic probes collect data for a portion (subgraph) of the directed graph of traders in a Turbonomic economy. The subgraph contains the relevant traders, which are the nodes of the directed subgraph.

Theoretically, it is possible to have probes with only one trader, generating trivial subgraphs with only one node. However, in typical instances, the subgraphs created by probes contain multiple traders. The subgraph in Figure 5 can be attributed to the data collected by a vCenter probe in a hypervisor environment.

Figure 5. The subgraph from a vCenter probe

The subgraph specification consisting of traders is built into the body of a probe code as a template. The template also contains the names of the commodities that the traders of the subgraph buy and sell. Along with the appropriate usage data (use, capacity, peak, and so on), other relevant attributes (active, resizable, and so on) can be associated with individual commodities. The specifications of all these types (the traders, the commodities, and the names of the commodity attributes) are buried into the probe body. Any change in the template regarding traders, commodities, or attributes needs a new image pull and consequent pod restart.

By invoking the relevant APIs of the underlying platforms with appropriate parameters, a probe determines the traders in the graph or subgraph. For each trader, it parses the results of the API invocation to determine the commodities traded with all their usage data and other attributes specified in the probe body and template.

The probe creates a data transfer object (DTO) from the data mentioned above for further processing. Figure 6 partially depicts the structure of a DTO created by a vCenter probe. Four traders exist in the DTO of Figure 6: the Data Center, the Physical Machine (Host), the Virtual Data Center, and the Virtual Machine (VM), all highlighted in yellow, trade in several commodities, a few of which are highlighted in green. The vCenter probe collects all the data related to the attributes of the traded commodities as specified in the template embedded in the probe code. The values of the attributes of the Power commodity sold by the Data Center trader and the CPU commodity sold by the Physical Machine (Host) trader collected by the vCenter probe are shown in Figure 6.

The actual DTO produced by a vCenter probe contains much more information, a lot of which is omitted for brevity in Figure 6, which is simply an attempt to portray the fundamental structure of a typical DTO constructed by a probe after completion of a round of data collection. The abridged DTO of Figure 6 corresponds to the middle portion of the directed graph of Figure 5, which is a proper subgraph of the entire economy of a Turbonomic deployment. The Data Center trader, a leaf node in the directed graph of Figure 5, does not buy any commodities in the DTO of Figure 6. Similarly, the Virtual Machine trader, the root node in the directed graph of Figure 5, does not sell any commodities in the DTO of Figure 6.

Figure 6. Sketch of a DTO returned by a vCenter probe

Probes, by default, gather all the required information corresponding to all the traders specified in the template and probe body at default or user-specified time intervals. However, a few probes have another mode of operation besides the default one. This mode of operation is called ‘incremental,’ which executes at much faster time intervals and generally gathers information only about a small number of attributes from a small number of selected traders of the subgraph specified in the template.

In incremental mode, the attributes to be gathered and the selected traders are specified in the probe body. It should be noted that the same probe code gets invoked for both default and incremental modes; a parameter controls the execution mode. The reason behind the incremental mode of execution of probes is to collect a small amount of selected data of interest from a small number of traders rapidly so that for the “selected data,” the probability of receiving stale information or the duration of staleness in the information received by the Optimization Engine, which is the ultimate consumer of the collected data, is significantly reduced. A dual-mode probe will also create a DTO in its incremental mode of execution. Typically, the incremental-mode DTO will be much smaller in size and complexity than the default-mode DTO for the same probe.

The vCenter probe is an example of a probe that has a dual mode of operation. In the default mode, the probe collects the relevant data about the subgraph of Figure 5 every ten minutes by default. However, the same vCenter probe also gets invoked in the incremental mode every thirty seconds by default to collect only the value of the “maintenance” attribute of the Host trader. Like the case of the default mode of probe operation, the invocation time interval for the incremental mode operation can also be changed using a Turbonomic Rest API. However, it should always be considerably less than that of the frequency of the default mode of operation to collect meaningful data.

Preprocessing collected data

The data that is collected by the probes are communicated in the form of DTOs to the Topology Processor module, which performs two critical activities before sending them out asynchronously for the use of the Optimization Engine: subgraph aggregation and data normalization.

In the Turbonomic architecture, the Topology Processor is the only component that directly interacts with probes. This article focuses on the Topology Processor's overall architecture and functionalities, not on its constituent modules' finer implementation details.

The Topology Processor merges the nodes of the same types in all the subgraphs. The number of merged nodes in a subgraph appears as the cardinality (weight) of a node in a directed graph of traders in Turbonomic. The aggregated and merged subgraphs from all the probes form the complete Turbonomic dynamic merged directed graph, the economy displayed in the Turbonomic GUI (see Figure 1 above).

Most of the Turbonomic deployments use more than one probe. However, theoretically, on-prem Turbonomic deployments with only one probe focusing mainly on infrastructure optimization might exist. The probe can be the vCenter probe, with possibly multiple copies associated with different targets for data collection in a deployment. A single probe Turbonomic deployment’s directed graph will be created from the probe's output without subgraph aggregation (joining).

Figure 7. The subgraph of an APM probe

For a Turbonomic deployment with multiple probes, the templates of probes are typically designed so that the directed subgraphs provided by individual probes have common endpoint traders. For example, we consider an on-prem Turbonomic deployment having two probes, APM and vCenter. The end node of the APM probe's directed subgraph is a Virtual Machine trader, as shown in Figure 7. The same Virtual Machine trader is the start node of the directed subgraph of the vCenter probe, as shown in Figure 5.

Figure 8. Aggregated subgraphs from an APM and a vCenter probe

Subgraphs obtained from the probes having common endpoint traders are aggregated (or joined) by the Topology Processor to a single graph. Aggregating the two directed subgraphs of Figures 7 and 5, the Topology Processor creates the joined directed graph of Figure 8. Proper aggregation of subgraphs by the Topology Processor is critical for the Turbonomic Optimization Engine to function. The Optimization Engine must know the complete present state of the economy to execute its optimization algorithms Turbonomic internally uses DTOs to represent, manipulate, and transmit graphs.

If connecting probes are missing or if we have a faulty probe definition (it is missing common start or end trader nodes in subgraphs), we might see an island of one or more traders that are not connected to the main Turbonomic directed graph. For example, in Figure 9, the Network trader is islanded because of missing connecting probes in the setup. In contrast, in Figure 1, corresponding to another environment, the Network trader node is not islanded but connected in the Turbonomic economy.

Figure 9. Islanded Network trader

Although without APM probes users can manually define the application subgraph portion of Turbonomic’s directed graph, a properly configured APM probe typically provides much more accuracy, detail, and dynamism in constructing the application subgraph. An APM probe will provide valuable information regarding application performance data needed for near real-time resource management.

An APM probe depends on the telemetry data sent by the APM utility. When there is no traffic on the application for a while, APM utilities might not generate any application monitoring data. Hence, the corresponding APM probes might not send any data to the Topology Processor. Without the application monitoring data, the Business Application trader node will appear islanded in the Turbonomic graph. Such is the case in Figure 10, where Instana is the APM utility, and all four applications monitored by Instana are idle for some time.

Figure 10. Islanded Business Application trader

The situation will change once applications get out of the idling state when they start receiving load. Instana will start generating monitoring data, and the Instana probe will start sending the data to the Topology Processor, which will create a graph with connected Business Application trader node as shown in Figure 1 for a different Turbonomic deployment.

The usage metrics collected by probes also undergo normalization in the Topology Processor module. Depending on the trader and the metric, Exponential Moving Averages (EMAs) or Percentile computations, are applied to the available historical data for data normalization. For example, EMAs are used for CPU or memory utilization for on-prem VMs, and percentiles are used for the same CPU and memory utilization for cloud VMs.

In abstraction, for the commodities whose usage value is in the probe templates, the Topology Processor provides the <normalized data, present utilization> tuples to the Optimization Engine for analysis. In addition to the Optimization Engine, the internal database consumes the same data by storing it in an archive. The Topology Processor uses these archived historical data to normalize the metrics the probes will collect in the future.

Several important communications between the Topology Processor, the probes, and the rest of the Turbonomic runtime are asynchronous. Asynchronous communication helps in scalability at the cost of architectural complexity. Figure 11 and the following description summarize the critical inter-component communications of Turbonomic at a high level in the context of data collection and pre-processing for an overall sound understanding, avoiding unnecessary implementation details.

Figure 11. Data collection and preprocessing

The probes get invoked asynchronously at a default or explicitly specified time interval. They send the parsed data to the Topology Processor for merger, aggregation, and normalization. Under normal working conditions, the Topology Processor always has the latest version of the complete Turbonomic graph in memory. The latest graph gets persisted in a local database so the Topology Processor can resume with the latest known Turbonomic graph on a restart. Whenever a probe sends a new DTO, default or incremental, the Topology Processor updates the portion of the graph corresponding to the DTO. Remember, the Turbonomic graph (and subgraphs) are dynamic. The Topology Processor also normalizes the utilization data present in the DTO. At a predetermined interval, called topologyBroadcastIntervalMinutes, whose default value is ten minutes, the Topology Processor places the entire set of processed information in a messaging queue, which in turn triggers the consumers of the queue, the Optimization Engine, and the archival module. On receiving the trigger, the Optimization Engine picks up the data from the queue for analysis, and the archival module inserts the data in a local database. The default value of topologyBroadcastIntervalMinutes, the data transmission interval from the Topology Processor, is suitable for most environments. Advanced Turbonomic users can alter the value of topologyBroadcastIntervalMinutes by invoking an internal Topology Processor method, and the new value will take effect without any restart.

Action execution

The Turbonomic Optimization Engine uses the data from the Topology Processor to generate actions. The probes execute the actions to bring and keep the system in the optimal state. Let’s now explore the Turbonomic action execution architecture and mechanisms and several subtle features of the action execution process.

Turbonomic Actions

In abstraction, a Turbonomic action is a tuple of two components: <an operation on the underlying platform, a set of participating traders>. For example, an on-prem VM vertical scale-up action can be abstracted as <a platform-specific VM resize-up operation, {the specific VM trader to be resized}>. Similarly, the tuple <a platform-specific VM move operation, {the source Host trader, the VM trader to redistribute, the destination Host trader}> is the abstraction of an on-prem VM redistribution action.

The Turbonomic modules that deal with actions often need to check whether two actions are equivalent. To make equivalence checking efficient, in addition to the complete action specification with all the attributes (components) necessary for its execution, Turbonomic internally also uses an integer representation of an action using a hash function on its relevant attributes. The hash value of an action is a large positive integer that can be up to 20 digits long and is called the Object Identity (OID) of the action. Using OIDs to represent actions trivializes determining the equivalence of actions to a simple equality check between two integers.

Using policies, Turbonomic users can schedule specific actions or categories of actions to get executed only at certain times, such as specific hours of the day, specified time periods in specified days of the week, and so on. Users typically restrict the execution period of selected varieties of actions, which can potentially result in non-responsive applications for a while, such as the potential application downtime is limited to non-peak or idle times.

The absence of any associated explicit scheduling policies with an action implies the action is scheduled for immediate execution. Policies also allow users to set the execution modes of categories of actions. There are several execution modes (manual, automatic, and so on) for actions that can be set using policies. The manual mode implies that the action can only be executed by Turbonomic when a user with appropriate authority approves the action. In contrast, Turbonomic will execute an action with automatic execution mode without requiring any user approval. The manual mode is also the default execution mode, which gets implicitly associated with an action without any associated policy specifying execution modes. An action can only be executed by Turbonomic when both its associated execution mode and scheduled time for execution, specified explicitly or otherwise, are satisfied. For details on scheduling actions and setting execution modes, see the “Working With Policies” section of the Turbonomic documentation.

Action execution mechanism

The actions generated by the Optimization Engine are executed by probes. In conformance with the Turbonomic architecture, like the process of metric data collection, only the Topology Processor component interacts directly with the probes for action executions.

In addition to the Topology Processor, the Action Orchestrator component of Turbonomic plays a critical role in coordinating the execution of actions.

In each execution cycle, the Optimization Engine places the generated actions, if any, in a messaging queue for the Action Orchestrator to process. On receiving actions, the Action Orchestrator processes them to select the actions that can be immediately executed while parking the others in temporary places. While doing so, the Action Orchestrator considers the execution schedules and modes of all the received actions, user-specified or default, and approval statuses when relevant. Conceptually, we can think of two internal queues: approval_queue and waiting_queue maintained and used by the Action Orchestrator. Actions that need approvals for executions are placed in the approval_queue, while actions that cannot be immediately executed because of their associated scheduling policies are placed in the waiting_queue.

Conceptually speaking, the Action Orchestrator goes to sleep after processing all the actions received from the Optimization Engine, and it wakes up again upon receiving freshly generated actions in the next or later execution cycle of the Optimization Engine. As part of processing the actions, the Action Orchestrator also checks whether any action in the approval_queue is now in the approved state. If so, and either it has no associated scheduling policy or its scheduling policy is satisfied, the action can be immediately executed; otherwise, it will be placed in the waiting_queue. The Action Orchestrator also traverses the waiting_queue to select actions whose associated scheduling policies are presently satisfied so that the actions are ready for immediate execution. The Action Orchestrator filters the immediately executable actions and places the rest in the two queues mentioned above in the order the actions were placed in the messaging queue by the Optimization Engine.

The Action Orchestrator passes the immediately executable actions to the Topology Processor for executions using the appropriate probes. The probes communicate the completion statuses of the actions they executed to the Action Orchestrator through the Topology Processor. Figure 12 schematically depicts the flow of action executions.

Figure 12. Action execution flow

An immediately executable action, depending on its type, resource requirements for its execution, and the present state of the economy (environment), can take some time to execute. Sometimes, the time can be significantly more than the execution cycle time interval of the Optimization Engine, which is typically ten minutes by default (see the earlier discussion of the topologyBroadcastIntervalMinutes attribute). It is quite possible that while an action on a set of traders is getting executed, the Optimization Engine generates the same action in each consequent execution cycle. These newly generated equivalent actions that either are getting executed or have just completed their execution will be essentially no-ops if executed and will serve no purpose besides possibly generating some confusing error messages regarding action execution failures. These redundant execution attempts might also consume some system resources.

To prevent this scenario of the attempt to execute the same action repeatedly, the Action Orchestrator maintains an internal executing_queue, which contains all the actions in the state of execution. Once an action arrives from the Optimization Engine and successfully passes the user-specified execution constraints, if any, the Action Orchestrator checks the presence of the action in the executing_queue. If it is present, the incoming action is discarded since its presence in the executing_queue implies that an identical action generated earlier by the Optimization Engine is already executing in the underlying system of the economy. Otherwise, the incoming action is placed in the executing_queue and is passed to the Topology Processor, which invokes an appropriate probe to execute the action. The OIDs of the actions, as mentioned earlier, are used by the action orchestrator to decide whether actions are identical. Once the execution of an action is completed, the corresponding probe passes the completed status information of the action to the Topology Processor, which in turn relays the information to the Action Orchestrator. On receiving the completed status information, the Action Orchestrator removes the action entry from the executing_queue. Now, if an action identical to the one just completed arrives from the Optimization Engine, and the user-specified timing constraints are satisfied, the action will be dispatched to the appropriate probe for execution.

Because actions typically require some amount of system resources (CPU, memory, disk, and so on) for executions, a system can be overwhelmed if too many actions start executing at the same time. This can stall the executions of some actions, or it might take a very long time to complete the executions of some of the actions. To prevent such a possibility, the Action Orchestrator limits the number of actions that can be executed simultaneously. The default number of concurrent actions submitted for executions is 1000, which should be good enough for most environments. A system administrator can change the value of the concurrentAutomaticActions configuration attribute in the Kubernetes yaml file for Turbonomic deployment (see the General Configuration Tasks section of the Turbonomic documentation). For a running Turbonomic instance, the change necessitates a restart of the Action Orchestrator pod. It should be evident that concurrentAutomaticActions is the maximum value of the depth of the executing_queue. If the number of actions in the action orchestrator that can be immediately executed exceeds the value of the above attribute, the actions will be enqueued in a pending_queue. As actions complete their executions and get removed from the executing_queue, new actions will be transferred from the pending_queue to the executing_queue.

A low value of the concurrentAutomaticActions attribute might unnecessarily keep many actions in the pending_queue even when the resources are available in the system for executing a larger number of concurrent actions. On the other hand, a very high value of the same attribute might overwhelm the system, resulting in very slow or even stalled executions of actions.

For proper action execution and coordination, the Action Orchestrator component conceptually uses several queues with possible transfer of one or more of the selected action elements from one queue to another depending on the status of action approval, scheduling time, and the availability of space in the execution_queue: from the approval_queue to the executing_queue, pending_queue, or waiting_queue; from the waiting_queue to the executing_queue or pending_queue; from the pending_queue to the executing_queue. The use of queues, which are a "first in, first out" data structure, ensures that the Action Orchestrator processes the action elements in the order of arrival from the Optimization Engine. Before enqueuing any action generated by the Optimization Engine in any of the internal queues of the Action Orchestrator, possible duplications are checked and discarded using OIDs.

During the processing of the newly arrived messages, the Action Orchestrator maintains the queues mentioned above so that only actions that are valid during the present state of the Turbonomic economy (the last execution cycle of the Optimization Engine) get considered for execution. The absence of any existing element of the approval_queue in the list of the newly generated actions immediately implies that the action is not valid in the present state of the Turbonomic economy. The Action Orchestrator discards such actions from the approval_queue.

The Action Orchestrator also traverses the waiting_queue to confirm the presence of the queue elements in the list of the newly generated actions, implying their validity in the present state of the economy and whose associated scheduling policies are presently satisfied. The Action Orchestrator promotes the waiting_queue elements for which both conditions are satisfied to the executing_queue or the pending_queue if the executing_queue is full.

For space efficiency, relevant Turbonomic components that deal with actions use “by-reference semantics.” Formally speaking, in a process space, actions are kept in a central location, and pointers (references) to them are passed around as needed for various initial data structures. Besides the executing_queue, the maximum queue depth for any other queue storing actions cannot be precisely determined. Theoretically, any of these queues might have to hold all the actions generated by the Optimization Engine in an execution cycle. Depending on the size and state of the economy and the user-defined constraints, the Optimization Engine can generate any number of actions. The Action Orchestrator persists the executing_queue and a few other internal data structures in a Turbonomic internal database to make itself restartable.

Probes provide metadata about the actions they execute. Based on that metadata, the Topology Processor selects a specific probe of the underlying platform to execute relevant actions. A probe uses the relevant attributes of the actions to invoke the appropriate API of the underlying platform to execute the intended action. During the action execution process, a probe communicates the status updates of the action execution to the Topology Processor at predetermined intervals, and the reporting interval is probe-specific.

Based on the underlying platform, the API invocations by the probes can be synchronous or asynchronous. For a synchronous API invocation, a platform communicates the progress of the action execution from the start of the process to the end state of success or failure, and the platform might also periodically update the probe about the actual amount of progress made.

The vCenter probe is an example of a platform that provides detailed status and progress information of its API invocations to the probe. On the other hand, the API invocation in the Kubernetes platform is asynchronous, a “fire-and-forget” style. In such cases, the corresponding probes are responsible for communicating the status information to the Topology Processor. The probes can do so by invoking platform APIs again to query the status of the execution.

At a minimum, such probes should communicate the “start” and the “result” of the API execution, success or failure, to the Topology Processor. On receiving the status information of the API invocations from the probes, the Topology Processor updates the status information of the executing actions in its internal data structures. Based on the topologyBroadcastIntervalMinutes predetermined time interval value, it communicates the statuses to the API module (see Figure 12), which is an internal component, for displaying the statuses in the Turbonomic GUI. On receiving the successful completion status information of an action from the corresponding probe, the Topology Processor, depending on the specifics of the action, might change the directed graph of the traders of the Turbonomic economy that it maintains, which will be communicated to the Optimization Engine within a time interval of topologyBroadcastIntervalMinutes. The Topology Processor communicates the completion status of actions to the Action Orchestrator for proper bookkeeping of the executing_queue.

Non-traditional action executions

So far, we have focused on the native traditional way of action executions, which many Turbonomic users use in their production environments. Besides the traditional probes, there are other ways to execute actions involving workflows or user-defined scripts. Let’s now review the execution of the two non-native styles of action execution.

ServiceNow probe

Actions whose execution mode was set by users to “Automated when Approved” will require ServiceNow, a popular ITSM software, for approval. Currently, Turbonomic works only with ServiceNow on the cloud, and to use it, users must download and install Turbonomic Actions in the cloud. The Turbonomic Actions is an application developed by Turbonomic and available in the ServiceNow store.

The Turbonomic Actions application acts as a gateway between Turbonomic and ServiceNow. It is important to note that ServiceNow is only for action approvals and not for action executions. Appropriate probes will execute the actions. Users can define a ServiceNow workflow embedding one or more Turbonomic actions for approval. The approval workflow can theoretically be simple, consisting of only one Turbonomic action itself. Or, it can also be somewhat complex, involving more than one Turbonomic action that needs approval and execution roughly at the same time. In many enterprise environments, all the essential actions are typically approved through ServiceNow. ServiceNow can also reject an action approval request. In that case, all subsequent approval requests for the equivalent actions get discarded by the Turbonomic Actions application for up to seven days by default.

Users can also audit the arrival of an action in the Action Orchestrator and the results of an action execution using internal tables of the Turbonomic Actions application.

Figure 13 schematically depicts the complete approval flow of actions associated with the ‘Automated when Approved’ execution mode in solid blue and eight steps.

Figure 13. ServiceNow approval flow of actions and audits

The Optimization Engine generates and sends the actions to the action orchestrator. Because of the specified execution mode, the Action Orchestrator sends the actions to the Topology Processor, which invokes the ServiceNow probe, which in turn passes the actions to the Turbonomic Actions application. The Turbonomic Actions application passes the actions to ServiceNow to create ‘change requests’ in ServiceNow for approval. At a point in time, ServiceNow can approve just one action or a group of more than one action waiting in the Turbonomic Actions application. The ServiceNow probe polls the Turbonomic Actions application to determine the status of the actions waiting for approval. The approved actions flow back to the Action Orchestrator through the Turbonomic Actions application, the ServiceNow probe, and the Topology Processor. The approved actions are ready for immediate execution, provided their associated scheduling policies, if any, are satisfied. The ServiceNow-approved actions whose associated scheduling policies prevent immediate executions are enqueued in the waiting_queue, like any actions associated with unsatisfied scheduling policies as discussed earlier. Their executions must wait until their scheduled times arrive. Once the immediately executable actions reach the Action Orchestrator component, the normal execution flow of the actions (as shown in Figure 12 above) starts.

If the arrival of a specific action in the Action Orchestrator is specified to be audited in the Turbonomic GUI, a similar but shorter flow originating from the Action Orchestrator and ending at the Turbonomic Actions application takes place (as shown in Figure 13 in the dotted blue line) in three steps. The audit flow for action arrival (often called “audit on generation”) might occur in parallel with the main approval flow. The Turbonomic Actions application logs a proper entry in its internal table for this audit.

If the completion of a specific action execution is specified to be audited, a very similar flow, like that of the “audit on generation,” occurs upon the completion of the action execution (as shown in Figure 13 in the dotted green line) in three steps. The result of the action execution, success, or failure is logged by the Turbonomic Actions application in another of its internal tables.

Both the above optional audits (“on generation” and “after execution”) apply to any action in general, provided the ServiceNow probe is configured in the environment. There is no need for the actions to be approved by ServiceNow or any other means. The ServiceNow probe never executes any action, as its primary purpose is to communicate with the Turbonomic Actions application for eventual action approval by ServiceNow.

Webhooks

Turbonomic’s Webhook feature can be used for advanced use cases for action execution, orchestration, and integration with other systems. A Webhook target and its corresponding probe are automatically created during Turbonomic’s installation. The Webhook target does not presently appear in the Turbonomic GUI. However, webhooks can be created, modified, and tested using the Workflow settings in the Turbonomic GUI.

Webhooks use the HTTP/HTTPS protocol to send data to the servers to execute custom scripts. To execute Turbonomic actions using Webhooks, users must use the properties of the Turbonomic ActionApiDTOs to create and send the proper payloads to the servers (see the DTO Index section of the Turbonomic documentation for more details).

Users can develop complex scripts to execute Turbonomic actions and any necessary orchestration workflows. A very simple orchestration can be to inspect and meet certain prerequirements in the environment before executing the intended action. For example, gracefully stopping an application before vertically scaling a Virtual Machine and sending a notification in the form of an email or a Slack message mentioning the success or failure after the execution of the scaling action.

A complex orchestration can be using an ITSM system that Turbonomic does not presently support. Webhooks can be designed and implemented to integrate specific Turbonomic actions into existing CI/CD pipelines. Turbonomic Webhooks are custom and they might require some effort to architect, develop, and test; however, they can be enormously powerful and flexible to meet specific needs for certain Turbonomic deployments. Turbonomic Webhooks can be considered a particularly flexible style of action execution, and they follow the standard scheme of action execution involving the Action Orchestrator, the Topology Processor, and the Webhook probe. For further details on Turbonomic Webhooks, see the Webhooks section of the Turbnomic documentation and this IBM AIOps & Management blog.

Conclusions

Turbonomic probes provide the environmental information to the Optimization Engine, the brain of Turbonomic, in the form of a directed graph of the traders and normalized resource usage data. The Optimization Engine analyses the information to generate actions to be executed on the underlying platforms to provide the optimal amount of resources to the applications.

Probes again play a critical role in executing the generated actions. In the Turbonomic eternal lifecycle of data collection, analysis, and action execution to bring and keep the underlying system at an desired state, probes provide the data collection and the action execution services. There are specialized probes dealing with the approvals for executing actions using ServiceNow and for action executions using user-developed scripts, which can potentially be very complex involving other IBM or third-party products.

A sound understanding of probes will help users deploy, configure, and use Turbonomic in environments with standard and specialized needs. Knowledge of probes can also be of value in real life in debugging Turbonomic, especially issues arising from the formation of Turbonomic’s economy and the control flows during data collection or action executions.

Turbonomic will continue to evolve, and it is not unreasonable to expect that it might integrate with other products in the future and expand its scope of optimization. The integration will be done through probes, perhaps specialized ones for specific products.

Acknowledgments

The authors gratefully acknowledge all the knowledge acquired during many fruitful discussions with several Turbonomic Engineers: Joseph Mate, Ron Even, Ory Valin, Nitya Vyas, Thirunavukkarasu Arunachalam, Zhuoran Bao, Apostolos Dailianas Ph. D., David Ohsie, Vivek Nandavanam, and Danilo Florisi Ph. D. Special thanks to Joseph for reviewing the draft quickly and providing suggestions for improvements.

The work started after a technical session with Christian Simony, Senior Infrastructure Architect, Nordea Bank. We realized the necessity of publishing a comprehensive technical article on Turbonomic probes. Thanks to Christian Simony and Derek Barclay of IBM, for all the interesting questions and agreeing to review the initial version.

Debasish acknowledges all the support and encouragement received during the work from Laura Scott, VP, and Aboud Ghazi. Andy Scheib sincerely supported and complemented Debasish in his endeavor to enhance and evangelize Turbonomic.

The authors also thank Michelle Corbin for her meticulous work in publishing this article quickly.