External Resource Manager Overview
InfoSphere Streams 4.0.1 introduced support for an external resource manager to supply resources for running InfoSphere Streams services and applications. This allows services and applications to run on resources other than traditional physical hosts, such as virtual machines or Linux containers. Use of an external resource manager also removes the need for users to tag resources. Alternatively, users associate a definition with a tag and the external resource manager allocates resources that match the definition.
Two domain configuration properties were added for external resource manager support:
- domain.externalResourceManager— enables an external resource manager.
Default: none – no external resource manager is enabled and the InfoSphere Streams resource manager is used.
Details: If specified, the value must match the type registered when the resource manager starts. Only one external resource manager can be specified, but the InfoSphere Streams resource manager is also used if there are resources with tags needed for an allocation.
- domain.managementResourceAllocationTags — contains the set of tags to use on a request for an external resource to run domain services.
Default: authentication,audit,jmx,sws – all domain services run on the resource.
Details: All domain services specified run on the same resource. Any domain service not specified results in a separate resource allocation to run the service. Any domain tag, reserved or user-defined, can be included. There is also a corresponding instance.managementResourceAllocationTags property for controlling instance service placement.
Tag handling has been enhanced for external resource management support as well. Definitions can now be associated with tags. A tag definition specifies properties required of resources having this tag. For example, a tag definition could specify a minimum memory requirement or a required software installation. The supported resource definition properties are defined by the external resource manager and can vary by manager. When a resource is requested from an external resource manager and tags are provided as part of the allocation request, the resource manager uses the tag definitions to determine the resource to return.
Once a resource is allocated by an external manager, InfoSphere Streams tags it with the tags specified on the allocation request. These tags either come from a managementResourceAllocationTags property or from an instance resource specification. Note this behavior is different than the tagging behavior for resources managed by the InfoSphere Streams resource manager. Users are still required to add tags to InfoSphere Streams resources in order to influence how they are chosen to run services and jobs. The reason why a user cannot tag an externally managed resource is because for some resource types, such as virtual machines, the resource may not even exist until an allocation request is made for it. The InfoSphere Streams resource manager also differs from external resource managers with tag definition handling. It does not support tag definitions; rather, it uses the tags that have already been added to the resources when determining which resources to use for running services or applications.
Developing an External Resource Manager
InfoSphere Streams provides an API for implementing an external resource manager. The API reference is part of the product documentation.
The primary requirement for implementing an external resource manager is to provide an implementation for the ResourceManager interface. The recommended way to do this is to extend the ResourceManagerAdapter class. ResourceManagerAdapter provides default implementations for many of the interface methods. It also shields external resource manager developers from future updates to the interface.
The ResourceManager interface includes methods for allocating and releasing resources. For allocation requests, relevant information about the client making the request, such as the architecture and operating system version, is included in the ClientInfo parameter so that a compatible resource can be returned. Tags can optionally be supplied in the AllocateInfo parameter to further identify the type of resource needed. The state of a returned resource can either be ResourceDescriptorState.AllocateState.ALLOCATED or ResourceDescriptorState.AllocateState.PENDING. A resource can be pending if, for example, it is in the process of being created or if it is waiting to be freed from another use. A ResourceManagerNotificationManager.resourceAllocated() notification must be sent when a pending resource becomes available. The InfoSphere Streams controller service must be started on the allocated resource and stopped when it is released. ResourceManagerUtilities provides utilities to help with this.
The ResourceManager.validateTags() method is called to verify the definition being associated with a tag is valid for the external resource manager. It may also be called to verify the definitions for a set of tags make up a valid combination for an allocation request. For example, if two tags are specified and one definition has a minimum memory requirement of 2 gigabytes and the other requires a minimum of 4 gigabytes, a resource satisfying both requirements could potentially be allocated. However, if one tag requires the resource to be located in a North American data center and the other tag requires a resource from a European data center, the tags would not be considered a valid combination because a resource could not satisfy both requirements.
The ResourceManager interface also includes methods for notifications and persistent data. Notifications are used to notify resource manager clients when a resource becomes available or when a resource is being revoked. The persistent data support provides a way for an external resource manager implementation to persist state data across restarts. It can also be used for recovery purposes in the event of an unexpected termination.
ResourceManager.customCommand() provides a way for clients of the external resource manager to invoke commands to access additional functionality provided by the resource manager implementation. For example, an external resource manager may define commands to query allocated resource information or query remaining capacity information. The supported commands as well as their request and response formats are defined by the external resource manager implementer. The InfoSphere Streams resource manager API also includes support for developing clients to call these commands. ClientConnectionFactory provides methods for establishing a connection to the resource server accepting resource manager requests and the returned ClientConnection object can be used to invoke the commands.
To enable an external resource manager, a ResourceServer object must be created and an instance of the external resource manager’s ResourceManager implementation class needs to be passed on the constructor. ResourceServer.start() must then be called. ResourceServer.start() registers the external resource manager’s type, host and port in the ZooKeeper repository. It then establishes a socket connection using the registered host and port to listen for resource manager requests. Any requests it receives are delegated to the ResourceManager implementation class that was passed on the constructor. Setting the domain.externalResourceManager property to the registered external resource manager type results in InfoSphere Streams allocation requests going to the corresponding ResourceServer.
A basic sample resource manager is included here. It can be used as a starting point for developing a custom resource manager implementation. This sample allocates resources using a static list containing resource names and attributes. All allocations are synchronous.
The sample resource manager supports two tag definition properties: cores and region. Both are optional. The cores property identifies the minimum number of processing cores required for a resource allocation request. The value must be an integer. If multiple tags contain this property, the largest value is used. The region property supports a finite set of values. If multiple tags include this property, all must have the same value; otherwise, the allocation request fails due to mutually exclusive tag definitions.
This sample resource manager implementation supports three custom commands: takeOffline, bringOnline, and queryResources. The takeOffline implementation shows an example of using the notification support. Sample client implementations which call these commands are also provided.
The readme.txt file contains a summary of the parts included in the sample. It also includes information on how to get started using the sample. More details about the sample implementation and external resource manager development in general are included as comments in the source parts.
In summary, support added to InfoSphere Streams 4.0.1 provides the ability to configure and use an external resource manager with InfoSphere Streams. This expands the types of resources that can be used to run services and applications. It also allows customers to take advantage of advanced capabilities provided by third-party resource managers. The resource manager API can be used to develop an external resource manager and the sample program provided with this article can be used to help get started with this process.