This document describes the step by step instructions to setup IBM Streams v4.1 to work with Platform Symphony resources.

Prerequisites :

Platform Symphony resource manager installation is not part of IBM Streams. Install Platform Symphony, Version 7.1, or later. For more details about Platform Symphony, please refer to the following website

http://www-03.ibm.com/systems/platformcomputing/products/symphony/

Create all the required resource groups and the consumers that will be used by IBM Streams.

Link to Platform Symphony Knowledge Center:

http://www-01.ibm.com/support/knowledgecenter/SSGSMK_7.1.0/sym_kc_welcome_71.html?lang=en

Procedure:

Install IBM¬†Streams 4.1.0.0 on a host that is not part of Platform Symphony cluster. Let’s call this host a client.

Instructions for setting up Streams 4.1.0.0 on symphony to shared file-system versus non-shared file-system are different. I will provide instructions for both these setups.

Setting up Streams 4.1.0.0 on Symphony to shared file system:

In this setup streams is installed to shared file system to which client machine and all the machines that are part of platform symphony cluster have access to the same Streams 4.1.0.0 installation.

1) To configure Streams environment, source streamsprofile.sh as shown below

$ source <Streams 4.1.0.0 install path>/bin/streamsprofile.sh

2) Update Platform Symphony properties files that are under $STREAMS_INSTALL/etc/symphony .

There are two properties files under $STREAMS_INSTALL/etc/symphony

consumer.properties
streams-symphony.properties

Update consumer.properties with the consumers as per your need. Please review all the available properties and customize the properties as per your need.

At the very minimum, update the following properties in streams-symphony.properties and set USER, PASSWORD, EGO_MASTER_LIST, EGO_KD_PORT. Please review all the properties and customize the properties as per your need.


# platform symphony credentials; values must be provided

USER=xxxxx

PASSWORD=xxxxx

# specifies the hosts that are master host candidates in the cluster (space separated)

EGO_MASTER_LIST=<symphony master hostname(s)>

# specifies the port number to use to connect to the Symphony cluster.

EGO_KD_PORT=<port>

# LogLevel values are: error, warn, info, debug, trace (default is error. I usually set it to trace)

LOG_LEVEL=trace

Setting up Streams 4.1.0.0 on Symphony to non-shared file-system:

In this setup we want to install streams to a non-shared file systems on all the Symphony hosts. Client machine and Platform symphony cluster do not share the same streams install. We have to use rsdeploy commands to install Streams on symphony cluster.

1) From the client machine where streams was installed earlier, create a resource package and this creates a tar file that should be deployed to symphony cluster using rsdeploy commands.

To create a resource package with default properties:


$ source <Streams 4.10.0 install path>/bin/streamsprofile.sh

$ $STREAMS_INSTALL/bin/streamtool mkresourcepkg

For a complete list of all the options and properties for the mkresourcepkg command, please refer to ‚Äústreamtool mkresourcepkg -h‚ÄĚ and ‚Äústreamtool man mkresourcepkg‚ÄĚ

2) Add resource package to Symphony repository and then install it on the resource groups as needed. These steps have to be performed from Platform Symphony master node.


$ ssh -X <symphony master node>

$ source /opt/ibm/platformsymphony/profile.platform

Following command add Streams resource package to platform Symphony cluster


$ rsdeploy add <package name> -p <path to resource package> -u <user> -x <password>

eg:

$ rsdeploy add Streams-4.1.0.0 -p ~/Downloads/StreamsResourceInstall-4.1.0.0-x86_64-el6.tar -u <user> -x <password>

Following command installs Streams resource package on all the hosts that are part of ComputeHosts resource group. We can specify multiple resource groups and installation will be done based on the requested resource groups.


$ rsdeploy install <package name> -c <consumer with root authority> - r <resource group> -u <user> -x <password>

eg:

$ rsdeploy install Streams-4.1.0.0 -c /InfoSphereStreams/streamsroot -r ComputeHosts -u <Admin user> -x <Admin password>

Following command can be used to check the status of Streams installation on the Symphony cluster. If the installation is successful, you will see the output as shown below. 


$ rsdeploy status <package name> -u <Admin user> -x <Admin password>

eg:

$ rsdeploy status Streams-4.1.0.0 -u <Admin user> -x <Admin password>

Sample output
HOST              STATUS       
symphonyhost1    INSTALLED    
symphonyhost2    INSTALLED    
symphonyhost3    INSTALLED 

3) Go back to Client node and update the properties files under <Streams 4.1.0.0 install>/etc/symphony as follows

Update consumer.properties with the consumers as per your need. Please review all the available properties and customize the properties as per your need.


At the very minimum, update the following properties in streams-symphony.properties. Please review all the properties and customize the properties as per your need.

# platform symphony credentials; values must be provided

USER=<Admin user>

PASSWORD=<Admin password>

# specifies the hosts that are master host candidates in the cluster (space separated)

EGO_MASTER_LIST=<Symphony master>

# specifies the port number to use to connect to the Symphony cluster.

EGO_KD_PORT=<symphony port>

# LogLevel values are: error, warn, info, debug, trace (default is error. I usually set it to trace)

LOG_LEVEL=trace

# non-shared file system; set to the install path on the resource

# example: RESOURCE_INSTALL_PATH=/opt/ibm/InfoSphere_Streams/4.1.0.0

RESOURCE_INSTALL_PATH=<Install path on the symphony resources>

Starting Streams-on-Symphony Resource Manager and Streams Domain:

The following steps are common to shared Streams install and non-shared Streams install.  To start a streams domain that uses the Symphony resource manager:

  1. Start  Streams-on-Symphony Resource Manager
  2. Start Streams domain, setting the domain.externalResourceManager as “symphony”

Following are steps on how to do this.

Starting Streams-on-Symphony Resource Manager

From the client machine, we can start streams-on-symphony in foreground mode and/or background mode. Default is background mode and the logs are created under /tmp/streams-on-symphony_$USER (as specified in streams-symphony.properties file). If run with –fg option the log messages are displayed in stdout .

NOTE: Setting up environment variables for STREAMS_DOMAIN_ID and STREAMS_ZKCONNECT will eliminate the need to pass ‚Äú-d <domain-id>‚ÄĚ and ‚Äú–zkconnect {<host>:<port>}‚ÄĚ with each streamtool command.


$ export STREAMS_DOMAIN_ID=<domain-id>

$ export STREAMS_ZKCONNECT=<ZooKeeper connection string>

Examples of streams-on-symphony start commands:

Following command starts streams-on-symphony for a specific domain within that zookeeper ensemble

$ streams-on-symphony start -d <domain-id> --zkconnect {<host>:<port>}

Following command starts streams-on-symphony that services multiple domain within that zookeeper ensemble

$ streams-on-symphony start --zkconnect {<host>:<port>}

Following command starts streams-on-symphony that services multiple domains in foreground mode

$ streams-on-symphony start --zkconnect {<host>:<port>} --fg

Following command starts streams-on-symphony that services a single domain in foreground mode:

$ streams-on-symphony start -d <domain-id> --zkconnect {<host>:<port>} --fg

Following command prints the status of all the streams-on-symphony resource managers

$ streams-on-symphony status --zkconnect {<host>:<port>} --all

Following command prints the status of the streams-on-symphony resource manager for a specific domain

$ streams-on-symphony status --zkconnect {<host>:<port>} -d <domain-id>

NOTE: Users can start multiple symphony resource managers. In that case one resource manager acts as a leader and others are in waiting state.

Starting Streams Domain Using External Resource Manager

Create a domain with domain.externalResourceManager=symphony

 $ streamtool mkdomain -d <domain-id> --zkconnect {<host>:<port>} --property domain.externalResourceManager=symphony 

For a complete list of all the options and properties for this command, please refer to ‚Äústreamtool mkdomain -h‚ÄĚ and ‚Äústreamtool man mkdomain‚ÄĚ

Start the domain as you normally do:

$ streamtool startdomain -d <domain-id> --zkconnect {<host>:<port>}

Start the domain and create instances as needed and submitjobs as needed.

Uninstall

If you need to uninstall this setup, below are the steps to clean up.

Stop all the domains and instances:

 $ streamtool stopinstance -d <domain-id> --zkconnect {<host>:<port>} 
 $ streamtool stopdomain -d <domain-id> --zkconnect {<host>:<port>} 

Stop resource managers by running the following command:

 $ streams-on-symphony stop -d <domain-id> --zkconnect {<host>:<port>} 

Remove the Streams domain:

 $ streamtool rmdomain -d <domain-id> --zkconnect {<host>:<port>} 

Uninstall Streams from Symphony cluster

 $ rsdeploy uninstall <package name> -c <consumer with root authority> -r <resource group> -u <Admin user> -x <Admin password> 

For example:

$ rsdeploy uninstall Streams-4.1.0.0 -c /InfoSphereStreams/streamsroot -r ComputeHosts -u <Admin user> -x <Admin password> 

Remove resource package from Symphony cluster. Please note that this step should not be performed before uninstalling the resource package from all the resource groups.

 $ rsdeploy remove <package name> -u <Admin user> -x <Admin password> 

For example:

$ rsdeploy remove Streams-4.1.0.0 -u <Admin user> -x <Admin password> 

Join The Discussion