Using the Streamtool job preview capabilities along with job configuration overlays enables you to tune your job configuration at submission time without changing the application code. This article illustrates the basic workflow for using some common configuration capabilities of job configuration overlays:
  • Persist job configuration parameters for reuse in multiple job submissions
  • Use the new streamtool previewsubmitjob command to learn about an application bundle’s characteristics
  • Adapt the host pool requirements of the application bundle to the instance configuration
  • Tune which operators are fused into which PEs, including user-defined regions of parallelism
  • Adjust the host placements for the operators by modifying the host pool definitions
See the “Job configuration overlays” link in the References section at the bottom to learn about the complete set of capabilities that are supported. This article is written from the perspective of a Streamtool command user. Console and Studio also provide similar capabilities with working with job configuration overlays. Their graphical presentation may provide some ease of use improvements, depending on how you normally interact with IBM Streams. See the References at the end of this document for details on those interfaces to see which approach is right for your needs.

Typical user scenario

To illustrate a typical work flow for using job configuration overlays, this article introduces a developer who has found a sample IBM Streams application that seems to meet many of the developer’s requirements. However, the sample application doesn’t match up perfectly with the developer’s environment constraints. This motivates the developer user to analyze the requirements, test the job, and adjust the job configuration settings, all without changing the application source code. To take advantage of these capabilities, the user performs the following steps:
  1. Acquiring the application bundle

    The user learns about an IBM Streams application called LogWatchUDP.sab. This SPL application watches the system messages file and flags security breaches, and includes user defined parallelism to prevent bottlenecks. The application seems perfect as a starting point for the user’s needs. After requesting and receiving the application bundle, the user finds a few usage-related notes with it that describe suggested changes to the deployment configuration- like which operators get fused into which PEs, and which resources the PEs get placed upon- to make it run optimally for his instance.
    LogWatchUDP usage notes:
    • Requires the following submission-time parameter
      file = specify the name of the log file to watch

    • The operators that are related to parsing the lines, which have the ParsedLines prefix in their names, are heavy resource users. If the instance has a host that is large enough to support them, it works best if they can be fused together and placed onto the large host.
  2. Analyzing the bundle requirements
    The user runs the previewsubmitjob command, specifying the LogWatchUDP.sab bundle. Running this command does a couple of things for the user. It produces a job configuration overlay file, called by default LogWatchUDP_config.json, that can provide a starting point for further configuration specification. It also performs a preview of the job submission that includes identification of some prerequisite type information that must be provided before a live submit job can be performed. It also provides the user with other useful information, such as the number of operators and PEs.
    From this information, the user can determine that the LogWatchUDP application contains twenty-seven operators that by default will be fused into five PEs for this user’s instance configuration. Note: if the user has large hosts in their instance, the number of PEs may be larger than five. All these PEs reference a host pool that is marked as exclusive, which means that these hosts are not able to be shared by any other jobs. This is a problem for this user, who needs to be able to share this hosts. Also, this application is configured to use an advanced host pool indexing technique to have fine-grain control over where the PEs get placed, which the user does not want to use.
    The previewsubmitjob section below describes in more detail how the user interprets the results to arrive at this information.
  3. Creating the job configuration overlay file for job submission
    As mentioned previously, the job configuration overlay file that is returned from the previewsubmitjob command can be used as starting template to specify other configuration options. The user copies the previewsubmitjob results file, LogWatchUDP_config.json, to the file that he will use for further submissions, in_config.json. While it can be used as is, it is strongly recommended to edit down the output JSON file for long-term purposes. For example, specify the minimal segments that contain the changes you want to make, and remove all the other sections. The extra information that is not needed for the subsequent input could make the input JSON file more susceptible to problems if the application bundle changes in the future.
    Tip: Try to keep the input JSON file as concise as possible. This will make it easier to work with, and make it more resilient to other changes.
  4. Persisting job configuration parameters
    To make it easier to specify the job parameters for each subsequent submission, the user decides to add the job configuration parameters to the in_config.json file.
    "jobConfig": {
      "jobName": "TheUsersJob",
      "dataDirectory": "/user/theUsers/dataDirectory",
      "submissionParameters": [
        {
          "name": "file",
          "value": "TheUsersData.log"
        }
      ]
    }
    
  5. Adapting the application’s host requirements to the target instance
    The user wants to make these changes to the host pool configuration:
    1. Remove the fine-grained host pool indexing and exclusivity that was specified in the bundle.
      ignoreBundleHostPlacements in the configInstructions section: Setting this to true resets all the host pool references to the default host pool with no tags, removes the exclusivity, and removes the in-pool indexing.
      "ignoreBundleHostPlacements": "true"
      
    2. Convert the default hostpool tagging with no tags defined in the bundle to the {rack1, rack2} tagging that exists in the user’s instance.
      convertTagSet in the configInstructions section: This setting converts the tagging requirement from the default host pool with no tags to {rack1, rack2}.
      "convertTagSet": [
        {
          "targetTagSet": [
            "rack1",
            "rack2"
          ]
        }
      ]
      
      Tip: If the "originalTagSet" is missing, it indicates that the default host pool that has no tags is the target of the conversion. If there was a tag set that was being converted from, then the "originalTagSet" clause is needed. 

For example, the setting for "originalTagSet": ["black", "yellow"] would convert every occurrence of black and yellow tags to rack1 and rack2 tags.
    3. Place the ParsedLines operators onto a host that has been tagged with BigHosts.
      poolPlacementGroups: This setting creates a new host pool for the BigHosts systems, and assigns all operators that begin with ParsedLines in their name to this host pool.
      "poolPlacementGroups": [
        {
          "name": "parsedLinePool",
          "operatorsInPool": [
            {
              "nameSpec": "ParsedLines*"
            }
          ],
          "tags": [
            "BigHosts"
          ]
        }
      ]
      
      Note: The use of the “*” wildcarding character means that this change affects both ParsedLines[*] and ParsedLinesMerged.out.
    4. Keep the number of PEs produced to the original number of five.
      Changing the above constraints may impact the number of PEs that are produced. To ensure that it remains at five, the following fragment is entered.
      "deploymentConfig": {
        "fusionScheme": "manual",
        "fusionTargetPeCount": 5
      }
      
  6. Optimizing operator fusion
    In keeping with the bundle author’s recommendations, the user now specifies that the ParsedLines operators get fused into same PE by creating a partitionColocationGroup in the in_config.json file.
    "partitionColocationGroups": [
      {
        "groupName": "ParsedLines_fused",
        "members": [
          "ParsedLines[*]"
        ]
      }
    ]
    
    If the user wants to further specify how operators within a parallel region are fused, the user may do this by specifying the following clause.
    "deploymentConfig": {
      "parallelRegionConfig": {
        "fusionType": "channelIsolation"
      }
    }
    
    This will cause all operators within a channel of a parallel region to only be fused together with other operators in that same channel. In this application, there are two parallel regions defined. This can be determined by noticing the operator whose names have brackets in the name, ParsedLines[*] and Failures[*].
    Note: Specifying this additional fusing constraint on the parallel regions may impact the number of PEs that are produced. Streams will attempt to produce five PEs as the user requested, but if this can not be met and still obey the channelIsolation constraint as well, then it will produce the minimal number of PEs that meets all the constraints.
    Note: In this example, the ParsedLines parallel region has specific partitionColocationGroups already specified for it, which will take precedence over this parallelRegionConfig configuration. All of the operators in all the channels of the ParsedLines[*] operators will be fused together due to the partitionColocationGroup configuration instead of being separated by channels as the parallelRegionConfig requests. If the user desires to remove the partitionColocationGroups that are defined in the bundle, they can do so by specifying the "ignoreBundleHostPlacements": partition clause in the configInstructions section.
  7. Testing the new job configuration overlay
    The user runs the streamtool previewsubmitjob command on the LogWatchUDP application bundle specifying the newly edited in_config.json. The user reviews the job configuration overlay file that is returned. If the user is happy with the results, the user saves this file away for future use. If the user is not, the user continues the cycle of editing and previewing the job submission until the user is satisfied with the results.
    streamtool previewsubmitjob -g in_config.json LogWatchUDP.sab
    
  8. Submitting the application bundle
    The user performs the job submission with the LogWatchUDP application, specifying the edited and tested input job configuration overlay file. The job deployment results match the user’s expectations from the job submission preview testing.
    streamtool submitjob -g in_config.json LogWatchUDP.sab
    
    Following these steps, the user was able to successfully adapt and deploy the application bundle into the user’s instance, with the deployment characteristics that the user needed.

Other hints when working with job configuration overlay files

Using wildcard characters in operator names

If there is a naming convention in the operator naming, using wildcard characters might make sense in operator configuration statements. The “*” character matches with 0 or more of any characters. Multiple “*“s can be specified in an operator name. One common use is to specify “*” within the [] of an operator name in a parallel region for configurations that should apply to the operators in all channels of a parallel region.

How to interpret results from the previewsubmitjob command

The previewsubmitjob operation is performed in two phases:
  1. Fusing the operators into PEs
  2. Placing the PEs on hosts
If any errors occur during the fusion phase, there will be one fusionError section similar to the following for every failed fusion attempt:
"fusionErrors": [
  {
    "detailedProblemMessages": [
      {
        "messageCode": "CDISR3204E",
        "substitutionTexts": [
          "ParsedLines_fused"
        ],
        "displayMessage": "CDISR3204E An operator with ParsedLines_fused partitionExlocation specified can not be fused with any other operator that also specifies it."
      }
    ],
    "names": [
      "ParsedLines[3]",
      "ParsedLines[4]",
      "ParsedLines[5]",
      "ParsedLines[6]",
      "ParsedLines[0]",
      "ParsedLines[1]",
      "ParsedLines[2]"
    ]
  }
]
The important fields to look at are:
  • "names": Names of the operators that are attempting to be fused
  • "displayMessage": Information on why the fusion attempt failed
If a fusionError occurred, the placement phase is not tried. Otherwise, the placement phase proceeds.
For each set of successfully fused operators, there will be a PlacementResult section like the following:
"placementResults": [
  {
    "peId": 3,
    "names": [
      "RawSuccesses",
      "Successes",
      "BreakinsWriter",
      "Breakins",
      "ParsedLinesMerged.Out"
    ],
    "tags": [
      "red",
      "blue"
    ],
    "hostPlacement": "HostA.myCompany.com"
  }
]
The "names" field lists the operators that will be fused into this PE. The "tags" field shows what the tagging requirements are for hosts that this PE can be placed on. The hostPlacement field indicates the host where this PE would be placed on if this were a real job submission, instead of a preview.
If the hostPlacement field contains NOT_PLACED, that means that a placement error occurred, and further information follows in the placementErrors section. The peId field can be used as a reference to further identify which placementErrors section refers to which placementResult entry.
"placementErrors": [
  {
    "placementNodeInfo": {
      "success": false,
      "placementMessage": {
        "messageCode": "CDISR3046E",
        "substitutionTexts": [
          "8:InFile,RawLinesFinal,RawLines,BreakinsWriter,ExecTime,RawSuccesses,ExecTimeWriter,Successes,Breakins"
        ],
        "displayMessage": "CDISR3046E The processing element ID 8:InFile,RawLinesFinal,RawLines,BreakinsWriter,ExecTime,RawSuccesses,ExecTimeWriter,Successes,Breakins cannot be scheduled. For more information, see the messages that follow this one."
      },
      "peOperatorSet": [
        "InFile",
        "RawSuccesses",
        "BreakinsWriter",
        "ExecTimeWriter",
        "Successes",
        "Breakins",
        "RawLinesFinal",
        "RawLines",
        "ExecTime"
      ],
      "peId": "8"
    },
    "contributingMessages": [
      {
        "messageCode": "CDISR3063I",
        "substitutionTexts": [
          "HostA.myCompany.com, HostB.myCompany.com, HostC.myCompany.com"
        ],
        "displayMessage": "CDISR3063I Because the following hosts already host a processing element, and therefore cannot host an operator that has a host isolation constraint, these hosts cannot be included in the scheduling process: HostA.myCompany.com, HostB.myCompany.com, HostC.myCompany.com."
      }
    ]
  }
]
Note: Sometimes when the configured constraints cannot be met, the placementError can be fairly complex. Streams Studio or Console may help present these errors in a more understandable format then reading the job configuration overlay JSON directly.

Legacy mode ignores fusion configurations

If legacy mode is in effect, either from directly being specified as the fusionScheme, or because the application bundle was compiled with a 4.1 or an earlier release, the configuration options related to fusion are ignored. See the “Job configuration overlays” reference for more details.

Ignoring bundle configuration information

Be aware that if you specify ignoreBundleRelationalPlacements or ignoreBundleHostPlacements, you are overriding some configuration information that the application’s author intentionally specified. Use these overrides with caution.

Working with job configuration overlay syntax and content

To keep the job configuration overlay as resilient to application changes as possible, keep the information in it to the minimal amount needed to express the minimal configuration that you changed. This particularly applies if you plan on using the output JSON file from the previewsubmitjob command. This output file contains a lot of extra information that is not needed for subsequent submissions.

JSON schema validation

As is typical for JSON-formatted files, there is no strict schema validation performed on the JSON file. If a keyword is in the wrong location, spelled wrong, or in the wrong case, it does not cause an error — it is simply ignored. One technique to check if an input JSON is correctly specified is to run the previewsubmitjob command and inspect the returned JSON file. If the input JSON file is also reported in the output JSON, then the input file was recognized correctly.

References

Join The Discussion