Resource monitors provide a mechanism to scan either a directory or a queue, and then submitting managed transfers once a specified trigger condition is met. If a monitor has been configured to:

  • Monitor a directory for files that match a specified pattern, and then transfer the files that match that pattern.
  • Monitor a queue on a queue manager for complete message groups, and then transfer those message groups.
  • Monitor a queue on a queue manager for individual messages, and then transfer the messages that are found.

then it is possible to overload the agent with managed transfers. In this blog post, we will look at how batching the files, message groups or individual messages into managed transfer requests can help prevent this.

The Scenario

Suppose we have a monitor that is configured to poll the directory C:\Input every 10 minutes looking for files that end in *.txt. When the monitor finds files that match this pattern, it submits a managed transfer to move those files to the directory C:\Output. Here is the task XML that the monitor will use when submitting the managed transfer requests:

<?xml version="1.0" encoding="UTF-8"?><request version="6.00" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="FileTransfer.xsd">
  <managedTransfer>
    <originator>
      <hostName>127.0.0.1</hostName>
      <userID>pault</userID>
    </originator>
    <sourceAgent QMgr=”QM1" agent="AGENT1"/>
    <destinationAgent QMgr="QM2" agent="AGENT2"/>
    <transferSet>
      <item checksumMethod="MD5" mode="binary">
        <source disposition="delete" recursive="false">
          <file>${FilePath}</file>
        </source>
        <destination exist="overwrite" type="file">
          <file>C:\Output\${FileName}</file>
        </destination>
      </item>
    </transferSet>
  </managedTransfer>
</request>

Notice the use of variable substitution here:

  • The source item contains the variable ${FilePath}, which will be resolved to the fully qualified name of the file that the monitor found.
  • The destination item contains a different variable – ${FileName}. When the monitor submits the task XML to the agent, it will replace this with the name of the file that met the trigger condition.

Now, suppose that five files are put into the C:\Input directory:

FileA.txt
FileB.txt
FileC.txt
FileD.txt
FileE.txt

What managed transfer requests will the monitor submit to the agent?

The default behaviour

If a monitor has not been configured with a batch size, then every file that matches the trigger pattern will result in a managed transfer being submitted to the agent.

In our example, there are 5 files that match the trigger (*.txt), and the task XML associated with the monitor will move the file that matches the trigger. This means that the monitor will submit five managed transfers (one for each file that matched the trigger), each containing a single item, which is the file that was found:

Managed Transfer

Transfer Item
1

FileA.txt
2

FileB.txt
3

FileC.txt
4

FileD.txt
5

FileE.txt

Depending on the configuration of the agent, these five managed transfers will run in parallel. This means that there is no guarantee that managed transfer 1 will complete before managed transfer 2.

Using a batch size of 2

Now, let’s assume that our monitor had been created with a batch size of two. This can be done by:

  • Either passing in the parameter -bs 2 into the fteCreateMonitor command.
  • Or by:
    • Selecting the “Batch together the file transfers, when multiple trigger files are found in one poll interval” checkbox.
    • And entering a value into the “Maximum batch size” box

    in either the “Create A New Resource Monitor” or “Edit An Existing Resource Monitor” wizards in the IBM MQ Explorer Managed File Transfer plugin.

When the monitor polls the directory, it finds all five files, splits them up into batches of two and submits three managed transfer requests to the agent:

Managed Transfer

Transfer Item
1

FileA.txt

FileB.txt
2

FileC.txt

FileD.txt
3

FileE.txt

Once again, these three managed transfers will probably run in parallel (depending on the configuration of the agent), which means that there is no guarantee the managed transfer 1 will complete before managed transfer 2.

Using a batch size of 5

Next, let’s reconfigure our monitor to have a batch size of 5. When it polls the directory, it will find all five files and group them up into a single managed transfer request:

Managed Transfer

Transfer Item
1

FileA.txt

FileB.txt

FileC.txt

FileD.txt

FileE.txt

In this case, the order that the files will arrive in the target directory (C:\Output) will be guaranteed. When the agent processes the managed transfer request, it will transfer FileA.txt first, followed by FileB.txt and so on.

Using a batch size of 10

Finally, let’s see what happens if we set the batch size to a value such as 10.

When the monitor polls the directory, it finds the 5 files. Even though this is less than the batch size, the monitor will still submit a single managed transfer to move these files. It does not have to wait until there are 10 files in the directory to start a managed transfer.

Managed Transfer

Transfer Item
1

FileA.txt

FileB.txt

FileC.txt

FileD.txt

FileE.txt

As always, I hope this helps! If you have any questions on this, feel free to ask and I’ll be happy to answer them.

Join The Discussion

Your email address will not be published.