Overview

Skill Level: Advanced

Working knowledge on IBM® MQ

This article demonstrates how to transfer Files / Messages using MQ Managed File Transfer across Enterprise Systems using an existing IBM® MQ network Infrastructure (both Cluster & Distributed).

Ingredients

IBM® MQ Managed File Transfer (MQMFT) transfers files between systems in a managed and auditable way, regardless of file size or the operating systems used. We are going to define some of the import concepts related MQMFT before proceeding with the actual implementation.

Concepts and Terminologies:

The Coordination, Agent, and Command Qmgrs are roles which runs on top of a given Qmgr(s). A single Qmgr can fulfill the role of all three, or you can have a queue manager dedicated to each role. The choice depends on your existing MQ infrastructure and the expected load on the network.

Agent
MQMFT agent is a JVM process that runs on a machine and performs file transfers to and from other agents. In order to transfer files to or from a machine using MQMFT, you must have an agent running on that machine. Every agent connects to a IBM MQ Qmgrs and uses MQ to communicate with other agents. The queue manager that an agent connects to fulfills the agent queue manager role, and will have a number of queues defined on it for the agent. An agent will connect only to its agent queue manager, which it does using bindings or client connection modes.

Agent Queue Manager
A Qmgr that hosts an Agent's queues. Each Agent needs a number of Queues defined to be able to function.

Coordination Queue Manager
A Qmgr that broadcasts audit of file transfer and acts as a central location.It also gathers information about all the agents in the network at a single point where it can be queried. All agent Qmgrs in the MQ network must have connectivity to the coordination queue manager to route appropriate information there.

Command Queue Manager
Submitting commands to an Agent (such as requesting a transfer) involves sending an MQ message to the agent's command queue on the Agent Qmgr. The Command Qmgr is the queue manager used by the command-line commands to connect to, in order to submit these command messages to Agents. IBM MQ routes the messages from the command queue manager to the appropriate Agent queue manager. The Command queue manager must have connectivity to all Agent Qmgrs.

How does IBM MQ Managed File Transfer work ? Read here.

The following Components need to be installed to be able use use MQMFT on Windows.

Install

MFT Packages for Linux Platform.

Install1

Step-by-step

  1. IBM® MQMFT over Cluster Network

    IBM® MQ v8.0 & Above

    Platform: Windows

    Introduction: In this article we are going to demonstate the implementation of MQMFT over an existing MQ Cluster Infrastructure. The below diagram represents our MQ Managed File Transfer Implementation for a given Organization having a Cluster ‘CLUS’ network. QM1 & QM2 are the Full Repository Qmgrs & the remaining Qmgrs are the Partial Repository. COLO (Coordination & Logger) is acting as the Coordination Qmgr for this MFT network. It also hosts the Logger function. The other four Qmgrs i.e. QM1, QM2, QM3 & QM4 have  been configured as the Command Qmgrs & also hosts the Agent Processes.

    We are going to use the below diagram as the Reference Architecture for our MQMFT Implementation. Using this scenario, we are going to send files across Agent Qmgrs in the cluster network.

    MQMFT-Clus

  2. Create a MQ Cluster

    In this Section, I are not going to create a MQ Cluster from scratch. Steps on how to create a MQ Cluster has already been mentioned in a separate but related article on IBM® MQ, an Enterprise Messaging Backbone in a True Sense! Hence, I am not going to repeat the Steps for Cluster creation & configurations here. This is how my system looks like after installation of MQMFT. The command to diplay the version of MQMFT is fteDisplayVersion as shown in the below screenshot.

    1-11

    We need to define a dedicated SVRCONN Channel through which the Coordination, Agent & Command processes are going to Interact with the respective Qmgrs in either Client or Binding mode. In our case, the Interaction will be in Binding mode because this entire exercise has been demonstrated in my local machine.

    runmqsc QM1
    #############
    DEFINE CHANNEL(MFT.SVRCONN)

    Similarly, repeat the same step for other four Qmgrs in the Cluster i.e. COLO, QM2, QM3 & QM4.

    Note: We have disabled the default MQ Security features in order to ensure no Security related errors while configuring the MFT. We will enable these security features once performed successfuly MFT configurations.

    ALTER QMGR CHLAUTH(DISABLED) ===> Disable MQ Channel Security (MQ v7.5 & above)

    ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)
    REFRESH SECURITY TYPE(CONNAUTH) ===> Disable ID/Password Security (MQ v8.0 & above)

    Final Setup Configuration: MQ Channels from QM1 (FR) Qmgr.

    QM1

    Final Setup Configuration: MQ Channels from QM2 (FR) Qmgr.

    QM2

  3. Setup the Coordination Qmgr (COLO)

    In this Section, we are going to setup the Coordination Queue Manager on top of COLO (PR) Qmgr as shown in the above Reference Architecture Diagram in Section 1.

    Generic Command: fteSetupCoordination -coordinationQMgr <QmgrName> -coordinationQMgrHost <Hostname> -coordinationQMgrPort <Port No.> -coordinationQMgrChannel <SVRCONN Channel>

    2-9

    We have run the COLO.mqsc file against the Qmgr to create the necessary Queue objects required by Coordination Process.

    Generic Command: runmqsc QmgrName <Qmgr.mqsc > Qmgr.mqsc.out

    3-8

    Location: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/config/COLO 

    Coordination

  4. Setup the Logger (COLO)

    Again, we are going to use the same Qmgr COLO (Coordination-Logger) to configure the Logger Process. Logger can be used as a DataBase or a Flat file to store information on historical account of file transfers. In our case, we have configured Logger to use a Flat file & limit its size to 5 MB.

    Generic Command: fteCreateLogger -loggerType FILE -loggerQMgr <QmgrName> -fileLoggerMode LINEAR -fileSize 5MB LOGGER_<QmgrName>

    Logger

    LOGGER_COLO_create.mqsc file has been created on successful completion of fteCreateLogger command. We have run this .mqsc file against the Qmgr to define the necessary MQ Objects required by Logger for its operations.

    Generic Command: runmqsc QmgrName <LOGGER_(Qmgr)_create.mqsc > LOGGER_(Qmgr)_create.mqsc.out

    Logger1

    Location: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/config/COLO/loggers/LOGGER_COLO

    Logger3

    Logger Commands
    ##################
    fteStartLogger LOGGER_COLO
    fteStopLogger  LOGGER_COLO
    ftsShowLoggerDetails

    Logger2

  5. Configure Agents & Command Qmgrs

    In this Section, we are going to create the Command properties file along with Agent Process for a given Qmgr QM1.

    Command 1: fteSetupCommands -connectionQMgr QM1 -connectionQMgrHost 127.0.0.1 -connectionQMgrPort 1414 -connectionQMgrChannel MFT.SVRCONN

    Command 2: fteCreateAgent -agentName AGENT_QM1 -agentQMgr QM1 -agentQMgrHost 127.0.0.1 -agentQMgrPort 1414 -agentQMgrChannel MFT.SVRCONN

    Command-Agent

    command.properties
    #####################
    #
    #Sat May 27 11:02:37 EDT 2017
    connectionQMgrChannel=MFT.SVRCONN
    connectionQMgrPort=1414
    connectionQMgrHost=127.0.0.1
    connectionQMgr=QM1

    ######################################

    Command-Agent1

    We have run the AGENT_QM1_create.mqsc file against the given Qmgr QM1 to create necessary MQ Objects required by Agent Process AGENT_QM1.

    Generic Command: runmqsc QmgrName <AGENT_(Qmgr)_create.mqsc > AGENT_<Qmgr>_create.mqsc.out

    Command-Agent2

    agent.properties
    ################
    #
    #Sat May 27 11:02:37 EDT 2017
    connectionQMgrChannel=MFT.SVRCONN
    connectionQMgrPort=1414
    connectionQMgrHost=127.0.0.1
    connectionQMgr=QM1
    enableQueueInputOutput=true

    ##############################

    Location: C:Program Files (x86)IBMWebSphere MQmqftconfigCOLOagentsAGENT_QM1

    Agent1-Properties

    Now, since the Command & Agent Process has been setup, we need to start the Agent using the command: fteStartAgent AGENT_QM1

    Start-Agent1

    Similarly, repeat the same Steps of Command & Agent Process configuration on other Qmgrs i.e. QM2, QM3 & QM4.

    Agent Commands
    #################
    fteStartAgent AGENT_QM1
    fteStopAgent  AGENT_QM1
    fteShowAgentDetails AGENT_QM1
    fteListAgents
    ftePingAgent -m QM1 -w 10 AGENT_QM1

    Once, all the Agent Processes has been configured for remaining Qmgrs in the Cluster except (COLO which is a dedicated Coordination & Logger Qmgr), list down the Agent processes using the below command: fteListAgents

    AgentStatus

    command.properties ==> For All Qmgrs in the Cluster.

    Command

    Now, it time to ping the FTE Agents in the Cluster network. Ensure that all the Cluster Channels are running before issuing FTE Ping command. Observer the response time in seconds marked in red.

    Agent-Ping1

    Another Output of ping FTE Agents from MQ Explorer. Ready Status signify that the Agent is ready to accept request for File transfer.

    Agent-Ping

    Log File Location for Agents & Logger Process: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/logs/COLO

    Logs

  6. File to FileTransfer

    In this Section, we are going to demonstrate the File Transfer (using MQ Explorer wizard) between various Qmgrs (Agents) in the Cluster network using the sequence of screenshots as shown below. We can also initiate the same transfer using Command Line Interface.

    1. Initiate a new File Transfer by Right Click on Managed File Transfer option as shown below & select New Transfer…

    New-Transfer1

    2. Select the Source & Destination Agents i.e. the Souce System from where you are going to pick up the file & the Destination System where you are going to send the file.

    New-Transfer2

    3. Click on Add button to select the Source system file & other associated transfer parameters required to complete File Transfer.

    New-Transfer3

    4. Here, select the various options as per your requirement for File Transfer. Some of the important parameters are marked in the below screenshot. In this case, we are going to transfer the file named POC.bat from the C drive of the system where AGENT_QM1 is running & place the same on the Desktop location of the other Agent AGENT_QM4.

    New-Transfer4

    5. Final Tranfer Configuration screenshot.

    New-Transfer5

    6.  You have the option of scheduling a Job to transfer files on specific date/time along with other variable configuration parameters as shown below.

    New-Transfer6

    7. The below screenshot show the file POC.bat has been successfully transferred from Souce to Destination Agents marked in Red.

    New-Transfer7

    Alternate Command: fteCreateTransfer -sa AGENT_QM1 -sm QM1 -da AGENT_QM4 -dm QM4 -df “C:UsersIBM_ADMINDesktopPOC.bat” “C:POC.bat”

  7. Message to File Transfer

    In this Section, we are going to put some messages in a Local Queue named POC defined under QM1. Using MQMFT, we are going to trasfer these 10 messages into a single File called TEST.txt to the Destination machine.

    1. Here, I have put 10 messages into the Local Queue POC.

    Msg-to-File2-1

    2.  In this Step, I have Selected the End point Agents i.e. Source & Destination be to AGENT_QM1 & AGENT_QM2. Selected the source to be Queue POC & the destination to a Text file by the name TEST.txt. As part of this operation, all the messages will be moved into the text file & place the same on Desktop.

    Msg-to-File1

    3. Final configuration screenshot before actual transfer.

    Msg-to-File2

    4. All 10 Messages has been successfully transferred as a Text file as shown below in the screenshot.

    Msg-to-File3-1

    5. Final output of the TEXT.txt file on the Desktop of the Destination Agent.

    Msg-to-File4-1

  8. File to Message Transfer

    In this Section, we are going to send a Text file TEST.txt as a Message to a Remote Agent Qmgr.

    1. Here, the Souce Agent is AGENT_QM4 & the Destination is AGENT_QM1. From the Souce machine, I am send a file named TEST.txt to a Local Queue POC on the Remote Agent machine as shown in the below screenshot.

    File-2-Msg

    2. The below screenshot shows the Text File has been successfully sent across machines & put into the Local Queue POC as message to remote agent (AGENT_QM1) Qmgr QM1.

    File-to-Msg-5

    3. The below screenshot show the souce file TEST.txt & its correspoding Message into the Local Queue POC.

    File-to-Msg4

  9. IBM® MQMFT over Distributed Network

    In this distributed setup, we are going to use the same Qmgr COLO as the Coordination & Logger Qmgr as created in Section 3 & 4 of this article. Hence, in this section we won’t repeat the steps but create the remaining MQ Connectivity configurations between the Qmgrs in this distributed network as per the below diagram. QMA, QMB, QMG and COLO are the four Qmgrs, used to demonstrate the File Transfer mechanism using MQMFT. COLO is dedicated as the Coordination & Logger Qmgr. Whereas the remaining three Qmgrs has been setup as the Agent & Command Qmgrs.

    As per the MQMFT requirement, all Agents must be registered with the COLO i.e. Coordination Qmgr in a MQMFT network. Hence, our first task would be to define normal Sender / Receiver Channels (Bi-directional) between COLO & other three Qmgrs i.e. QMA, QMB & QMG. Also, if we want to transfer files between Agent Qmgrs, you need to have MQ Channel connectivity in the first place. Hence, in the next section we are going to complete all the basic MQ configurations required for Managed File Transfer.

    QMA14

  10. Create a Distributed Network

    In this Section, I are not going to create a Distributed MQ Network from scratch. Steps on how to create a similar setup has already been mentioned in a separate but related article on IBM® MQ, an Enterprise Messaging Backbone in a True Sense! Hence, I am not going to repeat the same steps here. Here is what we have to do from MQ perspective:

    Step 1 (Required to register Agents with Coordination Qmgr)
    ################################################
    COLO <=== SDR / RCVR Channel ===> QMA
    COLO <=== SDR / RCVR Channel ===> QMB
    COLO <=== SDR / RCVR Channel ===> QMG

    Step 2 (Required to transfer files between Agent Qmgrs)
    #############################################
    QMA <=== SDR / RCVR Channel ===> QMB
    QMA <=== SDR / RCVR Channel ===> QMG
    QMB <=== SDR / RCVR Channel ===> QMG

    Besides, we need to define a dedicated SVRCONN Channel through which Agents & Commands processes are going to Interact with the respective Qmgrs in either Client or Binding mode. In our case, the Interaction will be in Binding mode because this entire exercise has been demonstrated in my local machine.

    runmqsc QMA
    ##############
    DEFINE CHANNEL(MFT.SVRCONN)

    Similarly, repeat the same step for other two Qmgrs in the network i.e. QMB & QMG. (Already completed this activity for COLO Qmgr in Section 2)

    Note: We have disabled the default MQ Security features in order to ensure no Security related errors while configuring the MFT. We will enable these security features once performed successfuly MFT configurations.

    ALTER QMGR CHLAUTH(DISABLED) ===> Disable MQ Channel Security (MQ v7.5 & above)

    ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)
    REFRESH SECURITY TYPE(CONNAUTH) ===> Disable ID/Password Security (MQ v8.0 & above)

    Final Setup Configuration: MQ Channels from COLO Qmgr

    QMA9

    Final Setup Configuration: MQ Channels from QMA Qmgr

    QMA7

  11. Configure Agents & Command Qmgrs

    Now, we are going to create the Command properties file along with Agent Process for Qmgr QMA.

    Command 1: fteSetupCommands -connectionQMgr QMA -connectionQMgrHost 127.0.0.1 -connectionQMgrPort 1430 -connectionQMgrChannel MFT.SVRCONN

    QMA

    Command 2: fteCreateAgent -agentName AGENT_QMA -agentQMgr QMA -agentQMgrHost 127.0.0.1 -agentQMgrPort 1430 -agentQMgrChannel MFT.SVRCONN

    Once the Agent has been created, we have run the AGENT_QMA_create.mqsc file against the given Qmgr QMA to create necessary MQ Objects required by Agent Process AGENT_QMA as shown below in the screenshot.

    Generic Command: runmqsc QmgrName <AGENT_(Qmgr)_create.mqsc > AGENT_<Qmgr>_create.mqsc.out

    QMA1

    Location: C:/Program Files (x86)/IBM/WebSphere MQ/mqft/config/COLO/agents/AGENT_QMA

    QMA10

    Similarly, repeat the same steps for Command & Agent process configuration on QMB & QMG Qmgrs.

    Agent Commands
    #################
    fteStartAgent AGENT_QMA
    fteStopAgent  AGENT_QMA
    fteShowAgentDetails AGENT_QMA
    fteListAgents
    ftePingAgent -m QM1 -w 10 AGENT_QMA

    Once, all the Agent Processes has been configured for remaining Qmgrs except (COLO which is a dedicated Coordination & Logger Qmgr), list down the Agent processes using the below command: fteListAgents

    ListAgents

    Verify the Agent Connectivity using Ping Command from MQ Explorer:

    QMA2

    command.properties ==> For All Qmgrs in the Distributed Network.

    QMA15

  12. File / Message Transfer

    Now, its time to send files across Agent Qmgrs within the Distributed Network. The procedure is exactly same as we have done in the Cluster setup. Select the Source & Destination Agents, Select File, Transfer Mode etc. as shown in the below screenshot. In this Case, I am going to send the file POC.bat from C drive of the Source Agent to Desktop of the Destination Agent as POC1.txt.

    1. File to File Transfer between AGENT_QMB & AGENT_QMA

    QMA4

    Click on the Finish button. If all the configurations are fine, then File transfer will be successful as shown in the below screeshot.

    QMA5

    2. Message to File Transfer: In this Case, I am putting a message into a Local Queue called POC123 defined under Qmgr QMA as shown in the below screenshot. Using the MFT Agents, I am going to send this message as a Text file to the destination system.

    QMA11

    Select the Source & Destination Agents. Select the Queue POC123 in the Souce Agent Qmgr, transfer type as Text transfer & specifiy the Destination path & file name POC123.txt as shown in the below screenshot.

    QMA12

     The below screenshot shows that the message has been successfully transferred to the Destination System as a Text file POC123.txt.

    QMA13

  13. Conclusion

    Thus, we have demonstrated the technical implementation of IBM® MQ Managed File Transfer (MQMFT) on top of an existing MQ Infrastructure either in the form of a Cluster or Distributed setup. MQMFT is one of the Advanced MQ features and hence it is essential to have some basic understanding of Agents, Coordination, Command & Logger components before proceeding with actual configuration. Since, MFT is basically using IBM® MQ as it underlying Infrastructure, we can apply all MQ Security features on top of MQMFT as well. For example, we can configure the MQ Channels to use SSL/TLS Certificates between Agent Qmgrs. In this article, I have used a series of related Screenshots as a step by step guide on how to configure MFT Agents, Coordination Qmgr, Logger etc. to simplify the process of MQMFT implementation.

    Additionally, you can checkout my other tech. articles on IBM® MQ:

    1. Advanced Clustering Techniques using IBM® MQ

    2. End to End Message Security using IBM® MQ

    3. Configuration of Multiple Certificates per Qmgr using IBM® MQ v8.0

    4. IBM® MQ, an Enterprise Messaging Backbone in a True Sense!

     

  14. Related Topics & References

    1. http://www-01.ibm.com/support/docview.wss?uid=swg27041613&aid=1

    2. http://www.mqug.org.uk/downloads/201307/201307%20-%20WMQ03%20-%20Introduction%20to%20WebSphere%20MQ%20Managed%20File%20Transfer.pdf

    3. http://mqtechconference.com/sessions_v2014/MQTC_Intro_t_%20MQ_MFT.pdf

    4. http://www-03.ibm.com/software/products/en/wmq-fte

    5. Managed File Transfer Solutions

    6. https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.wmqfte.doc/wmqfte_intro.htm

    7. https://www.ibm.com/developerworks/websphere/library/techarticles/1003_phillips/1003_phillips.html

4 comments on"Working with IBM® MQ Managed File Transfer (MQMFT)…"

  1. Hi Subhra-Mondal

    I have to configure a MQ cluster environment with multi-instance. Do you know if the procedure applies for that too?

    Thanks
    Akira

    • Subhra_Mondal October 13, 2018

      Hi Akira,

      I have not tested this configuration with Multi-instance Qmgr. However, I believe it will work as well since FTE is basically using underlying MQ architecture i.e. P2P or Cluster to send messages across network. Thanks!

      Regards
      Subhra

  2. pullarao atmakuri July 16, 2018

    Hi All,

    Found one issue for the above architecture.On one server we cannot have more than one command queue manager.As mentioned above each agent queue manager acting as a command queue manager.

    If it is on different servers solution looks good and if it is on one server we can have only one command queue manager only.

    • Subhra_Mondal October 13, 2018

      Hi Pullaroa,

      In this demo, I have configured multiple Command Queue Manager on the same machine. Please see the initial architecture each of the Qmgr is also acting as the Command QMGR (QM1-QM4). Thanks!

      Regards
      Subhra

Join The Discussion