IBM Streams Console V4.2 is now available! We incorporated feedback and suggestions from Streams customers and developers to provide a number of new enhancements and capabilities, notably: ZooKeeper enhancements for the monitoring of an ensemble, support for Kerberos authentication, integration with the Apache Edgent open-source project, the ability to securely store authentication information in an application configuration object and use it when interacting with Streams toolkits, improvements to the configuration of job submissions to optimize job deployment, the option of adding tag-host restrictions to control PE placement, and the addition of widgets to easily view log information.
ZooKeeper health and metrics monitoring
With V4.2, users can better monitor a ZooKeeper ensemble from the Management Dashboard. This may be helpful for troubleshooting purposes when Streams is unresponsive or slow due to poor ZooKeeper performance. The domain tree now displays the ZooKeeper ensemble nodes with status indicators for each node. Two new actions have also been added. The first is Check ZooKeeper Health, which allows you to easily monitor ZooKeeper performance by retrieving information on overall metrics (i.e., status, latencies, and outstanding requests) as well as metrics for operations (i.e., reads, writes, etc.). In addition, you can use the Reset ZooKeeper Statistics action to reset the ZooKeeper server and connection statistics.
For more information, please refer to this article.
Support for Kerberos authentication
IBM Streams now supports Kerberos as a method of authentication for enterprise domains, along with login modules and client certificates. Kerberos is a network authentication protocol developed by the Massachusetts Institute of Technology (MIT) that is designed to provide secure communications over a non-secure network using secret-key cryptography. The primary benefits are strong encryption and single sign-on (SSO). With Kerberos SSO, users can easily access the Streams Console without being prompted for their username and password credentials.
After creating a domain, a Streams administrator can configure Kerberos for Console by:
- Configuring the Kerberos encryption type for IBM Streams
- Configuring the Mozilla Firefox browser on Linux
- Setting up the Streams AAS and SWS services as the Kerberos service principals on all resources that are running the services
- Updating the IBM Streams domain properties that are used for Kerberos
- Obtaining and caching the Kerberos ticket-granting tickets that are used to authenticate user principals
For detailed setup instructions, please refer to this step-by-step tutorial.
Note that IBM Streams supports the following Kerberos V5 implementations and browsers: MIT Kerberos V5 Release 1.14 for Linux (with Mozilla Firefox), and Microsoft Active Directory for Windows Server 2012 (with Microsoft Internet Explorer). The Google Chrome browser is not supported.
Integration with Apache Edgent
Apache Edgent is an open source programming model and runtime environment for performing analytics on edge devices. An Apache Edgent application uses analytics to determine when data needs to be sent to a back-end system for further analysis, action, or storage. For example, you can use Apache Edgent to determine whether a system is running outside of normal parameters or a sensor picks up a reading that is outside of a specified threshold. To perform more complex analytics, you can connect the Apache Edgent application to your IBM Streams environment.
The Streams Console provides support for managing Apache Edgent applications and devices.
Application configurations and toolkits
Streams now allows application configuration information to be securely stored at the domain or instance level. These application configuration objects are useful to store information that toolkits can access dynamically at runtime. Toolkit operators have been updated to support retrieving connection information from application configuration objects alleviating the need to have connection documents on the job’s hosts. Refer to Streams Console 4.2: Secure Configuration Repository – RabbitMQ Example for details on how to use this new feature.
Configuring job submissions
In the previous version of Streams, jobs could be configured by specifying certain settings during submission-time, such as the data directory and PE trace setting. Streams V4.2 allows for even greater job customization to help improve run-time performance by expanding the number of configuration properties in four different categories: job properties, deployment configuration properties, operator configuration properties, and override instructions.
Besides being able to specify these submission-time parameters from the Console, Streams provides support for creating a job configuration overlay, a JSON file that contains name-value pairs for the submission-time parameters that you want to use. This file enables you to define, save, and distribute the submission-time parameters. Streams Console supports all the capabilities of this job configuration file such that you can import this file and adjust the existing settings as needed, as well as export this file for future use.
Furthermore, you have the option to preview how the submission-time configuration parameters that you specify will impact the runtime behavior of the application before submitting a job. The results will display the PEs that will be created, the hosts where each PE will be placed, and the tagging requirements. Errors will also be reported so that you can determine what constraints cannot be met, if any.
Using tags to restrict resources
Consider the following scenario where you have a Streams application in which a set of operators have been assigned to a host pool with a specified tag. Suppose you have a requirement such that no other operators should be run on the resources in that host pool. This can easily be achieved by creating tag restrictions between each host and the tag. To access this feature, open the Management Dashboard, hover over the Streams Tree menu icon , and click on Manage Resources and Tags. To create tag restrictions:
- On the left hand side, select the resource that you want to restrict
- On the right hand side, assign the custom tag to the resource (if not yet assigned) by checking the box and then click the lock icon to restrict the resource
- Repeat steps 1 and 2 as needed for each restriction
- Save the changes
- You will have the option to allow the tag restriction(s) even if the resources are part of an active instance. By clicking No, the operation to apply tag restrictions will fail if an instance is running on an impacted resource. By clicking Yes, the operation to apply tag restrictions will only fail if a PE is running on an impacted resource and does not satisy the tag requirement. Choose Yes or No.
Now, when you submit a job, only the PEs that refer to a host pool with a restricted tag are placed on the resource(s) in that pool.
Error log display
In order to better diagnose issues with the console retrieving information from the platform, a new log view was added that will show any errors or problems with internal data retrieval for the console. If you are experiencing problems with the console not updating correctly check this view for messages. It should be empty if things are running correctly. If there are messages in the view, contact IBM service and include the messages.
Here is the view with an error message:
See the following articles for details about the aforementioned new features:
- ZooKeeper Health and Metrics
- Configuring Kerberos authentication for Streamtool, Domain Manager, and Streams Console
- Configuring job submissions using Streams Console
- Securely store application specific information using the Secure Configuration Repository
IBM Knowledge Center links
- ZooKeeper serviceability enhancements
- Kerberos authentication support
- Apache Edgent integration
- Configuring submission-time job parameters
- Using tags to restrict resources
- Creating application configuration objects