This document:  Migration to Streams 4.0 for SPL Developer, explains the steps required to migrate an existing SPL application to Streams v4.  The Application Bundle feature also affects custom operators and toolkits, because it requires operators to handle files differently at compile time and runtime.  This document will describe the steps to migrate a custom operator to Streams V4.

Refer to this documentation for details on how to migrate a custom toolkit:  Migrating custom toolkits and operators

Index Toolkit

As a first step of migrating your toolkit, index your toolkit by running the following command:  spl-make-toolkit.  

Review Toolkit  Structure

Review all file resources that are part of your toolkit and determine if they are needed at compile time or runtime.  For files that are needed at runtime, make sure that they are placed in one of the directories that will be packaged as part of the application bundle.

If you have specialized directories that are not included into the application bundle by default.  You may specify these directories in the info.xml.

See this documentation for more detail:  Application bundle files

File-Related Parameters

As general best-practice, it is a good idea to follow these guidelines when designing a file-related parameter:

  1. The parameter should allow user to specify both absolute and relative path.
  2. If the parameter is for specifying a data file, the relative path should be relative to the data directory.
  3. If the parameter is for specifying a non-data file (e.g. configuration files), the relative path should be relative to the root of the application directory.

For each of the operators in your toolkit, identify all file-related parameters.  For each of the parameters, you need to determine the following:

  • Is the referenced file a data file or non-data file?
  • Does the parameter allow user to specify a relative path?  If a relative path is allowed, where is the path relative to.  Follow the aforementioned guidelines if possible.
  • Is the parameter value used at compile time or runtime?

Resolving Parameter Values at Compile-Time

If the parameter value is needed at compile time, the way to resolve the parameter value has changed.  Follow these steps to migrate:

1.  Constant to AttributeFree

We need to allow user to construct absolute file path in their SPL code using functions like this:

    file1:  dataDirectory() + [filename]
    file2:  getThisToolkitDir() + "/etc/config.cfg".

Therefore, a file-related parameter needs to be able to handle expression.  If the parameter was previously configured to be a Constant, you need to modify the parameter to be AttributeFree.

2.  Enable Expression Tree in Operator Model

If the parameter value is needed at compile time in a C++ primitive operators, you need to enable expression tree in the operator model.  This enables the SPL compiler to resolve the SPL expression at compile time, and would allow you to retrieve the value when the SPL application is compiled.

To enable expression tree, specify this in the operator context:

<splExpressionTree param="true"/>

This setting is enabled by default for Java operators.

3.  Use compileTimeExpression() to resolve value

Finally, to get the parameter value at compile time, instead of using getSplExpression() function, use the compileTimeExpression() function from the Perl module.

Here’s some example code:

my $rscriptParam = $model->getParameterByName("rScriptFileName"); 
if (defined $rscriptParam) {             
    my $splExpressionTree =  SPL::CodeGen::compileTimeExpression($model, $rscriptParam->getValueAt(0));

This code snippet, allows user to use code like this in SPL:

rScriptFileName : getThisToolkitDir() + "/impl/bin/process.r" ;

The compileTimeExpression(…) function will return the following value when the SPL application is compiled:


It is not possible to resolve parameter value at compile time for Java operators, therefore, no additional work is needed for Java operators.

4.  Construct absolute path from relative path

If the parameter value returned is a relative path, construct an absolute path from the relative path.

Example code that shows how this can be done at compile time:

# get "file" parameter Cpp expression from model
$::fileParam = $model->getParameterByName("theConfigFile");

# get the expression - converts getThisToolkitDir() to the full path to the toolkit directory
my $expression = SPL::CodeGen::compileTimeExpression($model, $::fileParam->getValueAt(0));
$expression =~ s/"(.*?)"/$1/s; # remove the beginning and end quotes

# create a path object
my $path = File::Spec->canonpath($expression);

# check if the path is relative
if(!File::Spec::Unix->file_name_is_absolute($path)) {
   # convert the relative path to an absolute path, the path is relative to the application directory
   $path = File::Spec->rel2abs($path, SPL::Operator::Instance::Context::getApplicationDirectory());

Resolving Parameter Values at Runtime

If the parameter value is needed at runtime, and relative path is allowed, the way to handle relative path has changed.  Your operator may rely on the fact that the current working directory is the data directory, and therefore all relative path was built relative to the data directory.

Follow these steps to migrate:

1.  Get parameter value at runtime

For C++ operator, you may resolve the parameter value using the following code snippet:

my $rscriptParam = $model->getParameterByName("rScriptFileName");
my $cppExpression = $rscriptParam->getValueAt(0)->getCppExpression();

For Java operator, you may resolve the parameter value using the following code snippet:


2.  Construct absolute path from relative path

Determine if the parameter value is a relative path.  If so, construct an absolute path from the relative path.  The root of the relative path is defined by you based on file type.  For example, if the file is a data file, the root of the relative path is the data directory.  If the file is a non-data file, the root of the relative path is the application directory.

Example code to construct the absolute path in C++:

#include <streams_boost/filesystem.hpp>

<% # get "file" parameter Cpp expression from model %>
<% $::fileParam= $model->getParameterByName("theConfigFile"); %>
<% $::fileParamCppValue = $::fileParam->getValueAt(0)->getCppExpression(); %>

// create the boost 'path' object
streams_boost::filesystem::path filePath(<%=$::fileParamCppValue%>);

// check if path is relative
if(filePath.is_relative()) {
   // if relative, convert to absolute path using the application directory as the base
   filePath = streams_boost::filesystem::absolute(filePath, getPE().getApplicationDirectory());

// check if file exists
if(!streams_boost::filesystem::exists(filePath)) {
   // handle error ...

// the boost 'path' class has the following 2 functions that may be useful:
//   string() - returns std::string
//   c_str() - returns const char *
loadFile(filePath.string(), ...);

Example code to construct the absolute path in Java:

// Use this function to get the path to a file relative to a given URI.
private String makeAbsPath(URI baseURI, String path)
            throws Exception {
        String toReturn = baseURI.resolve(path).toString();
        return toReturn;
// Create URI to base of relative path.  In this case, it is the application directory
URI configBaseDirURI = context.getPE().getApplicationDirectory().toURI();
// Call makeAbsPath to create URI based on application directory.
String absPath = makeAbsPath(configBaseDirURI, mypath.trim());

For details on how to access file paths at runtime, refer to this documentation:  Accessing paths at runtime

Join The Discussion