Automated High Availability and Next Generation ArchitectureInfoSphere Streams has had many high availability features for some time but in previous releases these could be complex to setup and require manual intervention in some cases to recover from failures. Streams V4 is self healing with automatic recovery from failures without manual intervention. The Streams V4 release includes a new next generation architecture allowing an administrator without any specialized HA skills to quickly and easily configure Streams to be resilient and use a single console to manage multiple instances with common users and hosts. Streams is now simpler to setup and administer, more resilient, more secure, more automatic and more dynamic.
New Streams Domain ConceptWe have introduced a new Streams domain concept which is a container for Streams instances which provides a single point for configuring and managing common resources, security and instances. A domain provides a single management console for monitoring and managing the domain and all of its instances. A domain manager helps users create and manage domains. The domain is responsible for the following:
- Configuration : Global configuration for the Streams domain and defaults for new instances.
- Instance management : Allow users to configure and manage instances.
- Resources : Allow users to configure and manage the host resources available for instances in the domain.
- Security : Users are configured and managed by the domain. The domain is responsible for authenticating users and checking that they are authorized to perform actions against the domain and instances.
- Public API : Provide JMX and REST apis to manage and monitor the domain and instances.
Faster, Simpler SetupThe Configuration of management service placement has been simplified using a tag based system. The user can allocate management services to resources using simple “management” and “application” system tags or let Streams place them fully automatically based on the resources available and the requested redundancy. More specific control is possible using system tags (audit, authentication, jmx, sws, view) for services which may require specific resources and firewall configurations. We have also removed dependencies so that shared file system and ssh are not required and DB2 is no longer used for recovery.
Improved Resiliency and Self HealingStreams Management services are now more resilient with automatic recovery from failures . Recovery is always on using zookeeper and support for multiple redundant copies of services removes single points of failure. Here are a few examples of how Streams handles failures automatically:
- Management Service Failure : A standby copy of the service takes over as the leader if the failed service was the leader. Restart the failed service.
- Management Host Resource Failure : Standby copy of services takes over as the leader if any services on the failed host resource were the leader. Restart the failed services on alternative resources. The failed Host resource will automatically rejoin the Streams domain and instance on recovery. Management services may automatically move back to the host resource or to newly added resources with appropriate tags to balance the load and increase resiliency to additional host resource failures.
- Application Host Resource Failure : Application PEs are automatically restarted on alternative resources.
All New Admin ConsoleThe Streams admin console has been completely updated with a new look and feel and customizable dashboard. A single console for each domain allows users to view all the resources, instances and jobs that they have permissions for. A summary of the system health is always visible at the top of the screen. This summary allows the user to filter what system objects are shown in the console and provides context based actions on the objects like “stop instance” for a running instance. A tree based view of all system objects similar to the Streams Studio explorer is also available. Dashboard widgets can be “flipped” for graphical and tabular views of the system data.
Simpler and Improved SecurityStreams users no longer require an Operating System user account on the Streams Resources and ssh is also no longer required. Streams can authenticate users against LDAP and then authorizes their actions based on the Streams permissions. The support for LDAP is more flexible to handle multi-part lookup and supports Microsoft Active Directory. Authentication and authorization checks are made for all api and tooling requests. Firewall configuration is also simplified with tag based placement of services which need to be accessed by tooling (jmx, sws, view) and configurable port ranges. Streams audit log capabilities have also been improved with a new audit service which uses configurable Log4j appenders to write the log. Audit log entries are created for all user commands and results including details like the Streams User, action (start instance, submit job etc.) the timestamp of the action and the command parameters.
Roles and Job GroupsRoles and Job Groups have been added to simplify managing permissions. User permissions can now be managed by user defined roles like “administrator” or “developer”. A role is a group of permissions that are managed by streams and are separate from user groups which are managed by LDAP or the OS. Permissions can be assigned to roles and then roles can be assigned to users or groups. Job permissions can now be managed by user defined Job Groups like “LogAnalytics” or “Ingest”. A Job is assigned to a job group when it is submitted for execution and permissions for that job are resolved based on the job group it is assigned to.
A Few More ThingsThe command line utility “streamtool” has been extensively updated to provide an interactive shell mode with command completion and history. This can be very useful to have open in a terminal or to pass more complex sets of commands using linux here documents. The install is now fully versioned which will simplify running multiple versions of Streams on the same cluster. Where Streams no longer requires a shared file system we have introduced application bundles and automatic provisioning to get management services and application code to the required resources to run. Applications now build to a self-contained relocatable bundle which can be submitted to a Streams instance for execution without any of the application code needing to already be on the Instance resources.
Application ResiliencyInfoSphere Streams has been able to detect Application Processing Element (PE) failures and automatically restart and reconnect the PE since its first release but this could result in tuples not being processed. We are introducing new consistent regions which allow a developer to guarantee that all data is processed in that region with a simple annotation (@consistent ) and HA compliant operators. Many systems for guaranteeing processing add an overhead to the processing of every tuple, we have taken a more lightweight approach where we periodically establish a consistent state that we can recover from on failure. A consistent state is a point in time where all tuples for all streams in a consistent region have been fully processed by the operators in the consistent region.
- Can reset their state and the state of any external system they interact with to the last consistent state on reset
- Can detect duplicates tuples being replayed since the last consistent state and do not process them again
- Are idempotent (tuples can be processed multiple times without changing the result beyond the initial processing of the tuple)