How does MFT resource monitor batching work?

How does MFT resource monitor batching work?

Paul Titheridge
Published on 25/04/2018 / Updated on 30/07/2018

Resource monitors provide a mechanism to scan either a directory or a queue, and then submitting managed transfers once a specified trigger condition is met. If a monitor has been configured to:

• Monitor a directory for files that match a specified pattern, and then transfer the files that match that pattern.
• Monitor a queue on a queue manager for complete message groups, and then transfer those message groups.
• Monitor a queue on a queue manager for individual messages, and then transfer the messages that are found.

then it is possible to overload the agent with managed transfers. In this blog post, we will look at how batching the files, message groups or individual messages into managed transfer requests can help prevent this.

The Scenario

Suppose we have a monitor that is configured to poll the directory C:\Input every 10 minutes looking for files that end in *.txt. When the monitor finds files that match this pattern, it submits a managed transfer to move those files to the directory C:\Output. Here is the task XML that the monitor will use when submitting the managed transfer requests:

<?xml version="1.0" encoding="UTF-8"?><request version="6.00" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="FileTransfer.xsd">

  <managedTransfer>

    <originator>

      <hostName>127.0.0.1</hostName>

      <userID>pault</userID>

    </originator>

    <sourceAgent QMgr=”QM1" agent="AGENT1"/>

    <destinationAgent QMgr="QM2" agent="AGENT2"/>

    <transferSet>

      <item checksumMethod="MD5" mode="binary">

        <source disposition="delete" recursive="false">

          <file>${FilePath}</file>

        </source>

        <destination exist="overwrite" type="file">

          <file>C:\Output\${FileName}</file>

        </destination>

      </item>

    </transferSet>

  </managedTransfer>

</request>

Notice the use of variable substitution here:

• The source item contains the variable ${FilePath}, which will be resolved to the fully qualified name of the file that the monitor found.
• The destination item contains a different variable – ${FileName}. When the monitor submits the task XML to the agent, it will replace this with the name of the file that met the trigger condition.

Now, suppose that five files are put into the C:\Input directory:

FileA.txt

FileB.txt

FileC.txt

FileD.txt

FileE.txt

What managed transfer requests will the monitor submit to the agent?

The default behaviour

If a monitor has not been configured with a batch size, then every file that matches the trigger pattern will result in a managed transfer being submitted to the agent.

In our example, there are 5 files that match the trigger (*.txt), and the task XML associated with the monitor will move the file that matches the trigger. This means that the monitor will submit five managed transfers (one for each file that matched the trigger), each containing a single item, which is the file that was found:

Depending on the configuration of the agent, these five managed transfers will run in parallel. This means that there is no guarantee that managed transfer 1 will complete before managed transfer 2.

Using a batch size of 2

Now, let’s assume that our monitor had been created with a batch size of two. This can be done by:

• Either passing in the parameter -bs 2 into the fteCreateMonitor command.
• Or by:
• Selecting the “Batch together the file transfers, when multiple trigger files are found in one poll interval” checkbox.
• And entering a value into the “Maximum batch size” box

in either the “Create A New Resource Monitor” or “Edit An Existing Resource Monitor” wizards in the IBM MQ Explorer Managed File Transfer plugin.

When the monitor polls the directory, it finds all five files, splits them up into batches of two and submits three managed transfer requests to the agent:

Once again, these three managed transfers will probably run in parallel (depending on the configuration of the agent), which means that there is no guarantee the managed transfer 1 will complete before managed transfer 2.

Using a batch size of 5

Next, let’s reconfigure our monitor to have a batch size of 5. When it polls the directory, it will find all five files and group them up into a single managed transfer request:

In this case, the order that the files will arrive in the target directory (C:\Output) will be guaranteed. When the agent processes the managed transfer request, it will transfer FileA.txt first, followed by FileB.txt and so on.

Using a batch size of 10

Finally, let’s see what happens if we set the batch size to a value such as 10.

When the monitor polls the directory, it finds the 5 files. Even though this is less than the batch size, the monitor will still submit a single managed transfer to move these files. It does not have to wait until there are 10 files in the directory to start a managed transfer.

As always, I hope this helps! If you have any questions on this, feel free to ask and I’ll be happy to answer them.

6 comments on"How does MFT resource monitor batching work?"

1. April 12, 2019

Hi Paul,

I have one query: Suppose in my Managed transfer there are two Item transfer to be present.One is for placing file in a Queue and another is the same file to be archived in some file location. And i have given bs as 50. i have placed 40 messages in source location. But when the transaction starts it is showing 80 transfers have initiated in a single batch. and transfers are stuck in Starting phase.

Can you help me on this how to set up batch size for this kind of scenario.

Thanks,

• Paul_Titheridge April 15, 2019

Thanks for the question. It sounds like the managed transfer started by the resource monitor consists of two steps:

Step 1: Copy the contents of the source file into a message on a queue.
Step 2: Once step 1 has completed successfully, move the source file to another location for archiving.

If the resource monitor has been configured with a batch size of 50, then it should perform these two steps for a maximum of 50 files within a single managed transfer. For example:

******************************************************
Managed Transfer 1:
————————-
Step 1: Copy the contents of File1 into a message on a queue.
Step 2: Move File1 into the archive location.

Step 1: Copy the contents of File2 into a message on a queue.
Step 2: Move File2 into the archive location.
……
Step 1: Copy the contents of File50 into a message on a queue.
Step 2: Move File50 into the archive location.
******************************************************
Managed Transfer 2:
————————-
Step 1: Copy the contents of File51 into a message on a queue.
Step 2: Move File51 into the archive location.
…
******************************************************
and so on.

Based on what you have said, I suspect that the Task XML containing the definition of the managed transfer to be started by the resource monitor isn’t quite right. Rather than batching up the files into a single managed transfer, it is starting a single managed transfer for each file (and possibly including multiple files in each managed transfer). I’d suggest reviewing the Task XML, and checking that the source filenames for the two steps mentioned above are specified using the ${FilePath} substitution variable. More information on variable substitution can be found here:
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.adm.doc/variable_substitution.htm

Hope this helps! If you still have issues, I’d recommend raising a PMR so that myself and my colleagues in IBM Support can take a look.

Paul

1. September 14, 2018

Hi Paul,
thanks for this article but what about using a resource monitor to monitor a queue? In our company we are using MFT only for Message-To-File and File-To-Message but it seems that setting the batch size only works for File-To-Message.

I put 100 msgs on a queue and created a monitor which is polling the queue with -bs 10. In filelogger I see three entries (TSTR, TPRO, TCOM) for each message that has been transferred to the filesystem. I don’t see any difference when setting batch size or not.

Any advices?

Best regards,

• Paul_Titheridge September 18, 2018

Thanks for reading the blog post. I hope you found it useful.

Batching should work for message-to-file transfers too. However, there have been a number of issues with it which have prevented it from working as expected. As a first step, I would recommend that you upgrade your MFT installations to the latest Fix Pack, to pick up all of the necessary fixes in this area. If you still have issues, then raise a PMR, and myself and my colleagues in IBM Support can take a look.

Thanks

Paul

1. August 07, 2018

Does the file logger log the batch or still each individual file are logged as without the batch?

• Paul_Titheridge August 08, 2018

Thanks for reading this blog post. I hope you found it useful!

In answer to your question, the file logger will record information about each of the managed transfers that are processed by an agent. If we consider the examples above, this means:

– If the batch size is not set, then the file logger will record information about 5 different managed transfers, each containing a single item.

– If the batch size is set to two, then the file logger will record information about three managed transfers:
– Managed Transfer 1, which contains File A and File B.
– Managed Transfer 2, which contains File C and File D
– Managed Transfer 3, which just contains File E.

– If the batch size is set to 5, then the file logger will record details of a single managed transfer, which contains all 5 items.

I hope this helps! If you have any additional questions on this, let me know and I’ll be happy to answer them.

Paul