Overview

Skill Level: Advanced

IBM MQ, FTE and IBM Integration Bus v10.0

IBM MQ Advanced Managed File Transfer integrates with an IBM integration Bus to Process files that have been transferred by IBM MQ Advanced Managed File Transfer as part of an IBM Integration Bus flow or other way round.

Ingredients

Required Skills

Familiarity with the IBM MQ File Transfer.
Good understanding on IBM Integration Bus Administration
IBM Integration Bus development skills. ESQL requires for manipulating message content and other headers. This article is not intended to show any payload transformation.
Good Knowledge on IBM MQ.

System Requirements

IBM MQ Advanced v9.0
IBM Integration Bus v10.0
Linux

Note: IBM MQ Advanced on distributed platforms gives your enterprise entitlement to install Managed File Transfer Service. Also, Unlimited installs of Managed File Transfer Agent, providing the Queue Manager the Agent connects to is entitled using IBM MQ Advanced capacity.

Step-by-step

  1. Introduction

    From IIBv10, IBM MQ is no longer required for using IBM Integration Bus. However, FTEInput and FTEOutput nodes require IBM MQ Server to be installed on the same machine as integration node running, and they are available for use only if you specify a queue manager on the integration node. Integration Node should be created by using mqsicreatebroker with -q option.

    By choosing any of the following options, Integration Bus receive or send files to a destination on MQ File Transfer network.

    • Send files to an existing IBM MQ Advanced Managed File Transfer network from an IBM Integration Bus network: In this scenario, MQ input node receive the messages from IBM MQ infrastructure, then it further process to FTEOutput Node by generating a file. The destination agent places the file at destination file directory mentioned in FTEOutput Node.
    • Receive files from an existing IBM MQ Advanced Managed File Transfer network to an IBM Integration Bus network: In this scenario, FTEInput node receive the file from MQFT agent, then it further process to MQ Output node to place the messages in Queues. FTEInput & FTEOutput nodes should have read and write permission on Input and Output directories.
  2. System Setup – Naming Conventions

    In this article, Below Queue Managers has been setup.

    COQM – Coordination Queue Manager

    AGQM – Agent Queue Manager

    CDQM – Command Queue Manager

    IIB_FTE – Queue Manager to run Integration Node, acts as an agent Queue Manager for Integration Node

    The below Figure 1, shows the File Transfer between the IBM MQ v9.0 & IBM Integration Bus v10.0 in a Linux Machine

     

     

                          Figure-1

                                                                                            

            Figure 1 , shows the File Transfer between the IBM MQ v9.0 & IBM Integration Bus v10.0 in a Linux Machine

  3. Setup the following Message Channels between your Queue Managers:

    • From the agent queue manager to the coordination queue manager
    • From the command queue manager to the agent queue manager.
    • From the agent queue manager to the command queue manager
    • From the command queue manager to the coordination queue manager
    • From the agent queue manager to any other agent queue manager in the IBM MQ Advanced Managed File Transfer network

     

     

                     Figure-2

     

     Figure 2, shows the File Transfer between IBM MQ Advanced Managed File Transfer & IBM Integration Bus, it shows the architectural representation of how the channels are connected between the Agent Queue Manager, Coordination Queue Manager, Command Queue Manager, Agents on the IBM Integration Bus.

     

    Notes:

    • Ensure that IBM MQ is installed on the same machine as your integration node.
    • Information about file transfers is held on storage queues that are controlled by IBM MQ, so you must install IBM MQ on the same computer as your integration node if you wish to use the capabilities provided by the FTEInput and FTEOutput nodes.
    • As recommended by IBM, you must try not to use the integration node queue manager as a coordination queue manager
  4. Creation of MQ Objects Required for Managed File Transfer Connectivity on a Linux Server

    We will first start with the detailed steps for the creation of MQ Objects that are required for the Managed File Transfer Connectivity on a Linux Server:

     
    1) Check the version of IBM MQ using the below given Command:

    bash-4.2$ dspmqver

    2) Output displayed for the Version of MQ

                            Name: IBM MQ

                            Version: 9.0.0.0

                             Level: p900-L160512.4.TRAIL

                            Platform: IBM MQ for Linux (x86-64 platform)

                            Mode: 64-bit

                            O/S: Linux 3.10.0-514.10.2.el7.x86_64

                            InstName: Installation1

                            InstDesc:

                            Primary: Yes

                            InstPath: /opt/mqm

                            DataPath: /var/mqm

                            MaxCmdLevel: 900

    3) Create the COQM,AGQM,CDQM and IIB_FTE Queue Managers:                

                            crtmqm COQM

                            crtmqm AGQM

                            crtmqm CDQM

                            crtmqm IIB_FTE

    Notes:

    Queue Manager IIB_FTE is required for Integration node to run and acts as an Agent Queue Manager

    4) Start the COQM,AGQM,CDQM and IIB_FTE Queue Managers:

                            strmqm COQM

                            strmqm AGQM

                            strmqm CDQM

                             strmqm IIB_FTE

    5) Check the status of the Queue Managers:

    bash-4.2$ dspmq

    6) Creation of the Queue Managers are successful & Status: Running:

    QMNAME(COQM) STATUS(Running)

    QMNAME(AGQM) STATUS(Running)

    QMNAME(CDQM) STATUS(Running)

    QMNAME(IIB_FTE) STATUS(Running)

  5. Creation of Required Objects Under the COQM Queue Manager

    We will now look at the detailed steps for the creation of MQ Objects that are required under the Queue Manager COQM

     
    1) runmqsc Command for Queue Manager COQM:

    bash-4.2$ runmqsc COQM

    2) Define the Listener:

    DEFINE LISTENER(LIS.COQM) TRPTYPE(TCP) PORT(7770) CONTROL(QMGR)

    Output displayed for the Listener Defined:

    AMQ8626: WebSphere MQ listener created.

    3) Start the listener:

    START LISTENER(LIS.COQM)

     Output displayed after Starting the Listener:

    AMQ8021: Request to start WebSphere MQ listener accepted.

    4) Define the Channels required:

    DEFINE CHANNEL(AGQM.COQM) CHLTYPE(RCVR)

    DEFINE CHANNEL(CDQM.COQM) CHLTYPE(RCVR)

    DEFINE CHANNEL(IIB_FTE.COQM) CHLTYPE(RCVR)

    Output displayed after the Channels are Defined:

    AMQ8014: WebSphere MQ channel created.

     5) Go to IBM MQ Installation Directory bin Location:

    In this example location is: /opt/mqm/bin

     Configuring the Coordination Queue Manager:

    ./fteSetupCoordination -coordinationQMgr COQM -f

     
    Output displayed after the Coordination Queue Manager Configuration:

    BFGCM0243I: A file has been created that contains the MQSC definitions for your coordination queue manager. The file can be found here: ‘/var/mqm/mqft/config/COQM/COQM.mqsc’.

     After running the fteSetupCoordination command, run the coordination_qmgr_name.mqsc script from the MQ_DATA_PATH/mqft/config/coordination_qmgr_name directory to perform the necessary configuration for the Coordination Queue Manager:

    In this Example:

    MQ_DATA_PATH/mqft/config/coordination_qmgr_name = /var/mqm/mqft/config/COQM/COQM.mqsc

    6) Run COQM.mqsc against COQM Queue Manager

    runmqsc COQM < /var/mqm/mqft/config/COQM/COQM.mqsc

     Output displayed for the MQSC Commands given:

    All valid MQSC commands were processed.

     
    Notes: If the Coordination Queue Manager has been started with strmqm -c, the command deletes the changes made above and makes it default. In that case, run the coordination_qmgr_name.mqsc script on the queue manager again.

  6. Creation of Required Objects Under the AGQM Queue Manager

    We will now look at the detailed steps for the creation of MQ Objects that are required under the Queue Manager AGQM

    1) Define the Listener:

    DEFINE LISTENER(LIS.AGQM) TRPTYPE(TCP) PORT(7771) CONTROL(QMGR)

     Output displayed for the Listener Defined:

    AMQ8626: WebSphere MQ listener created.

     2) Start the listener:

    START LISTENER(LIS.AGQM)

     Output displayed after Starting the Listener:

    AMQ8021: Request to start WebSphere MQ listener accepted.

    3) Define the Channels required:

    DEFINE CHANNEL(AGQM.COQM) CHLTYPE(SDR) CONNAME(‘localhost(7770)’) XMITQ(COQM)

    DEFINE CHANNEL(AGQM.CDQM) CHLTYPE(SDR) CONNAME(‘localhost(7772)’) XMITQ(CDQM)

    DEFINE CHANNEL(AGQM.IIB_FTE) CHLTYPE(SDR) CONNAME(‘localhost(7773)’) XMITQ(IIB_FTE)

    DEFINE CHANNEL(CDQM.AGQM) CHLTYPE(RCVR)

    DEFINE CHANNEL(IIB_FTE.AGQM) CHLTYPE(RCVR)

     Output displayed after the Channels are Defined:

    AMQ8014: WebSphere MQ channel created.

     5) Define the Queues for Enabling Channel Triggering:

    DEFINE QL(COQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(AGQM.COQM)

    DEFINE QL(CDQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(AGQM.CDQM)

    DEFINE QL(IIB_FTE) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(IIB_FTE)

     Output displayed after the Queues are Defined:

    AMQ8006: WebSphere MQ queue created.

     6) Configuring the Agent Queue Manager:

    ./fteCreateAgent -agentName AGQM -agentQMgr AGQM -p COQM -f

     Notes: fteCreateAgent command creates an agent and its associated configuration.

     
    Output displayed:

    BFGCM0238I: Direct the following MQSC definitions for agent ‘AGQM’ to queue manager ‘AGQM’.

    BFGCM0239I: A file has been created containing the MQSC definitions to define the agent AGQM. The file can be found here: ‘/var/mqm/mqft/config/COQM/agents/AGQM/AGQM_create.mqsc’.

    BFGCM0241I: A file has been created containing the MQSC definitions to delete the agent AGQM. The file can be found here: ‘/var/mqm/mqft/config/COQM/agents/AGQM/AGQM_delete.mqsc’.

    BFGCL0053I: Agent configured and registered successfully.

    Notes:

    These queues are internal system queues that you must not modify, delete, or read messages from unless you are deleting the agent.

     MQSC Commands issued are available in a file from the following location:

     MQ_DATA_PATH\mqft\config\coordination_qmgr_name\agents\agent_name\agent_name_create.mqsc

     In this example,

     MQ_DATA_PATH\mqft\config\coordination_qmgr_name\agents\agent_name\ = /var/mqm/mqft/config/COQM/agents/AGQM

     Run AGQM_create.mqsc against AGQM:

    7) runmqsc AGQM < /var/mqm/mqft/config/COQM/agents/AGQM/AGQM_create.mqsc

     Output displayed for the MQSC Commands given:

    All valid MQSC commands were processed.

  7. Creation of Required Objects Under the CDQM Queue Manager

    We will now look at the detailed steps for the creation of MQ Objects that are required under the Queue Manager CDQM

    1)  Define the Listener:

    DEFINE LISTENER(LIS.CDQM) TRPTYPE(TCP) PORT(7772) CONTROL(QMGR)

    Output displayed for the Listener Defined:

    AMQ8626: WebSphere MQ listener created.

    2) Start the listener:

    START LISTENER(LIS.CDQM)

     Output displayed after Starting the Listener:

    AMQ8021: Request to start WebSphere MQ listener accepted.

     3) Define the Channels required:

    DEFINE CHANNEL(AGQM.CDQM) CHLTYPE(RCVR)

    DEFINE CHANNEL(IIB_FTE.CDQM) CHLTYPE(RCVR)

    DEFINE CHANNEL(CDQM.AGQM) CHLTYPE(SDR) CONNAME(‘localhost(7771)’) XMITQ(AGQM)

    DEFINE CHANNEL(CDQM.COQM) CHLTYPE(SDR) CONNAME(‘localhost(7770)’) XMITQ(COQM)

    DEFINE CHANNEL( CDQM.IIB_FTE) CHLTYPE(SDR) CONNAME(‘localhost(7773)’) XMITQ(IIB_FTE)

    Output displayed after the Channels are Defined:

    AMQ8014: WebSphere MQ channel created.

    4) Define the Queues for Enabling Channel Triggering:

    DEFINE QL(AGQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(CDQM.AGQM)

    DEFINE QL(COQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(CDQM.COQM)

    DEFINE QL(IIB_FTE) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(CDQM.IIB_FTE)

     Output displayed after the Queues are Defined:

    AMQ8006: WebSphere MQ queue created.

     5) Configuring the Command Queue Manager:

    ./fteSetupCommands -connectionQMgr CDQM -p COQM -f

     Notes: This command creates the command.properties file in the coordination queue manager configuration directory /var/mqm/mqft/config/COQM.

     ls -lrt command.properties

    -rwxrwx–x. 1 mqm mqm 52 Apr 3 14:13 command.properties

     Ensure that you have already created and configured a coordination queue manager before you issue the fteSetupCommands command.

     You see the following output:

     BFGCL0245I: The file ‘/var/mqm/mqft/config/COQM/command.properties’ has been created successfully.

     The command queue manager does not require extra IBM® MQ definitions. After you run fteSetupCommands, the command.properties file is created in the configuration directory.

  8. Starting IBM MQ Managed File Transfer Agent

    Steps for Starting the IBM MQ Managed File Transfer Agent

     1) Start the Managed File Transfer Agent by using the fteStartAgent command. We should start the agent before performing the file transfer:

    ./fteStartAgent -p COQM AGQM

     Output displayed after Starting the Managed File Transfer Agent:

    5655-MFT, 5724-H72 Copyright IBM Corp. 2008, 2014. ALL RIGHTS RESERVED

     BFGCL0030I: The request to start agent ‘AGQM’ on this machine has been submitted.

     BFGCL0031I: Agent log files located at: /var/mqm/mqft/logs/COQM/agents/AGQM/logs

    the agent’s work load.

     [15/03/2017 20:01:26:346 IST] 00000001 AgentRuntime I BFGAG0058I: The agent has successfully initialized.

     [15/03/2017 20:01:26:657 IST] 00000001 AgentRuntime I BFGAG0059I: The agent has been successfully started.

     Navigate to logs and check for Agent status. Output displayed:

    13/04/2017 11:52:43:731 EDT] 00000001 AgentRuntime I BFGAG0058I: The agent has successfully initialized.

     [13/04/2017 11:52:44:120 EDT] 00000001 AgentRuntime I BFGAG0059I: The agent has been successfully started.

  9. Stopping IBM MQ Managed File Transfer Agent

    Steps for Stopping the IBM MQ Managed File Transfer Agent

    Stop the Managed File Transfer Agent:

    -bash-4.2$ ./fteStopAgent -p COQM -i AGQM

    Output displayed after Stopping the Managed File Transfer Agent:

    5655-MFT, 5724-H72 Copyright IBM Corp. 2008, 2016. ALL RIGHTS RESERVED

    BFGCL0467I: Issuing immediate stop request to agent ‘AGQM’. The command will wait for the agent to stop.

    BFGCL0465I: Agent ‘AGQM’ has been stopped.

    Notes: BFGAG0125W: The maximum size to which the java heap can grow is ‘512’MB, which is the default value. This value may be too low dependent on the agent’s work load. (This warning occurs every time, when agent stops/starts).

    For an instance, we have set the value to 1024. Execute the below command to set the value. It varies based on the system configuration and business requirement.

     set FTE_JVM_PROPERTIES=”-Xmx1024M”

    After set this value to 1024, error is disappeared from

     /var/mqm/mqft/logs/COQM/agents/AGQM/logs.

  10. IBM Integration Bus Configurations

    IBM Integration Bus Configurations

     We will now look at the detailed steps required to configure all the different properties for the Broker

     1) Create an Integration Node:

    mqsicreatebroker IIB_FTE -q IIB_FTE

    Output displayed after Creating the Integration Node:

    BIP8071I: Successful command completion.

    2) Start the Integration Node:

    mqsistart IIB_FTE

     Output displayed after Starting the Integration Node:

    BIP8096I: Successful command initiation

    3) Create an Integration Server:

    mqsicreateexecutiongroup IIB_FTE -e IIB_FTENode

     Output displayed after Creating an Integration Server

    BIP1124I: Creating integration server ‘IIB_FTENode’ on integration node ‘IIB_FTE’…

    BIP1117I: The integration server was created successfully.

     Output displayed after the Integration Node has initialized the integration Server.

    BIP8071I: Successful command completion.

  11. Creation of Default System Queues

    IBM Integration bus require access to IBM MQ components to server File transfer capabilities. These capabilities require a set of default system queues to be created on the queue manager that is associated with the integration node. These system queues are used to store information about in-flight messages.

    We will now look at the detailed steps for the creation of Default System Queues on a Linux machine.

    Navigate to Integration Bus Installation Directory: install_dir/server/sample/wmq

     In this example, directory is /opt/mqsi/iib-10.0.0.1/server/sample/wmq

     1) Creation of Queues:

    ./iib_createqueues.sh IIB_FTE mqm

     Notes: The iib_createqueues.sh is a Script File that is shipped along with the IBM Integration Bus Product Installation.

     Output displayed after Creation of Queues

    AMQ8006: WebSphere MQ queue created.

    26 MQSC commands read.

    No commands have a syntax error.

    All valid MQSC commands were processed.

    The setmqaut command completed successfully.

  12. Creation of Required Objects Under the IIB_FTE Queue Manager

    We will now look at the detailed steps for the creation of MQ objects that are required under the Queue Manager IIB_FTE

     1) Define the Listener:

    DEFINE LISTENER(LIS.IIB_FTE) TRPTYPE(TCP) PORT(7773) CONTROL(QMGR)

     Output displayed for the Listener Defined:

    AMQ8626: WebSphere MQ listener created.

    2) Start the listener:

    START LISTENER(LIS.IIB_FTE)

     Output displayed after Starting the Listener:

    AMQ8021: Request to start WebSphere MQ listener accepted.

     3) Define the Channels required:

    DEFINE CHANNEL(IIB_FTE.AGQM) CHLTYPE(SDR) CONNAME(‘localhost(7771)’) XMITQ(AGQM)

    DEFINE CHANNEL(IIB_FTE.COQM) CHLTYPE(SDR) CONNAME(‘localhost(7770)’) XMITQ(COQM)

    DEFINE CHANNEL(IIB_FTE.CDQM) CHLTYPE(SDR) CONNAME(‘localhost(7772)’) XMITQ(COQM)

    DEFINE CHANNEL(CDQM.IIB_FTE) CHLTYPE(RCVR)

    DEFINE CHANNEL(AGQM.IIB_FTE) CHLTYPE(RCVR)

     Output displayed after the Channels are Defined:

    AMQ8014: WebSphere MQ channel created.

     4) Define the Queues for Enabling Channel Triggering:

    DEFINE QL(AGQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(IIB_FTE.AGQM)

    DEFINE QL(COQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(IIB_FTE.COQM)

    DEFINE QL(CDQM) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) USAGE(XMITQ) TRIGDATA(IIB_FTE.CDQM)

    DEFINE QL(MQInput.QL)

    DEFINE QL(MQOutput.QL)

    Output displayed after the Queues are Defined:

    AMQ8006: WebSphere MQ queue created.

  13. Creation of Queues Required for Managed File Transfer Agent

    We will now look at the steps for the Creation of the Queues required for the Managed File Transfer Agent on the Integration Node Queue Manager IIB_FTE

    1) Refer to the Attachment called “Queues for the Managed File Transfer Agent.mqsc”, This will be used in the section below to replace the Default Agent Name (SYSTEM.FTE.*******.Agent name) with the required Agent Name (IIB_FTE.IIB_FTENODE )

    DEFINE QLOCAL(SYSTEM.FTE.COMMAND.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(5000) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.DATA.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(5000) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.REPLY.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(5000) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.STATE.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(5000) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.EVENT.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(5000) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.AUTHAGT1.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(0) + MAXMSGL(0) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE DEFINE QLOCAL(SYSTEM.FTE.AUTHTRN1.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(0) + MAXMSGL(0) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.AUTHOPS1.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(0) + MAXMSGL(0) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE DEFINE QLOCAL(SYSTEM.FTE.AUTHSCH1.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(0) + MAXMSGL(0) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.AUTHMON1.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(0) + MAXMSGL(0) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.AUTHADM1.IIB_FTE.IIB_FTENODE) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(0) + MAXMSGL(0) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE DEFINE TOPIC(‘SYSTEM.FTE’) TOPICSTR(‘SYSTEM.FTE’) REPLACE

    ALTER TOPIC(‘SYSTEM.FTE’) NPMSGDLV(ALLAVAIL) PMSGDLV(ALLAVAIL) DEFINE QLOCAL(SYSTEM.FTE) LIKE(SYSTEM.BROKER.DEFAULT.STREAM) REPLACE

    ALTER QLOCAL(SYSTEM.FTE) DESCR(‘Stream for WMQFTE Pub/Sub interface’) * Altering namelist: SYSTEM.QPUBSUB.QUEUE.NAMELIST * Value prior to alteration: DISPLAY NAMELIST(SYSTEM.QPUBSUB.QUEUE.NAMELIST)

    ALTER NAMELIST(SYSTEM.QPUBSUB.QUEUE.NAMELIST) + NAMES(SYSTEM.BROKER.DEFAULT.STREAM+ ,SYSTEM.BROKER.ADMIN.STREAM,SYSTEM.FTE)

    * Altering PSMODE.

    * Value prior to alteration

    DISPLAY QMGR PSMODE

    ALTER QMGR PSMODE(ENABLED)

    DEFINE QLOCAL(SYSTEM.FTE.DATABASELOGGER.REJECT) + DESCR(‘Messages rejected by the FTE database logger.’) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(999999999) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

    DEFINE QLOCAL(SYSTEM.FTE.DATABASELOGGER.COMMAND) + DESCR(‘Command messages to control the FTE database logger.’) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(999999999) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(5000) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE  

     Copty the above MQSC definitions to a file called “Queues for the Managed File Transfer Agent”.

    Notes: The agent name is derived from IntegrationNode.IntegrationServer, and is not configurable

    2) File should be executed by using the following command

    runmqsc IIB_FTE < “Queues for the FTE Agent.mqsc”

  14. Determining the Agent Name

    We will now look at the alternative steps for Determining the Agent Name

    After successful creation of Agent, the message is written to IIB logs. Please refer below log.

    IBM Integration Bus v10001 (IIB_FTE.IIB_FTENode) [Thread 8217] (Msg 1/1) BIP3358I: The WebSphere MQ File Transfer Edition agent ‘IIB_FTE.IIB_FTENODE’ in integration server ‘IIB_FTENode’ on integration node ‘IIB_FTE’ has been started. It is registered with the coordinating queue manager IIB_FTE and will use the configuration stored in /var/mqsi/components/IIB_FTE/4ae3cc1b-af7e-401e-9cfe-6d05bbbd4192/config/WMQFTE/config

    Notes: From the logs, we can see that the ‘IIB_FTE.IIB_FTENODE’ is the Agent Name we arrive at.

     Also, as an alternative we can identify the Agent Name by issuing mqsireportproperties command:

     1) mqsireportproperties IIB_FTE -e IIB_FTENode -o FTEAgent -n agentName

    Output displayed:

    IIB_FTE.IIB_FTENODE

    Notes: By default, the agent uses the queue manager that is specified on the integration node as the coordination queue manager. If the queue manager specified on the integration node is being used as the coordination queue manager, the integration node configures it as a coordination queue manager.

     

    A warning is written to the log if the coordination queue manager is not changed from the default. Please refer below log:

    IBM Integration Bus v10001 (IIB_FTE.IIB_FTENode) [Thread 8217] (Msg 1/1) BIP3360I: A temporary WebSphere MQ File Transfer Edition agent called ‘IIB_FTE.IIB_FTENODE’ in integration server ‘IIB_FTENode’ on integration node ‘IIB_FTE’ has been created

    Always use different queue manager as the coordination queue manager. Set the coordination queue manager to COQM for integration server IIB_FTE in integration node IIB_FTENode by executing the below command:

    2) mqsichangeproperties IIB_FTE -e IIB_FTENode -o FTEAgent -n coordinationQMgr -v COQM

    Output displayed:

    BIP8071I: Successful command completion.

    After running this command, you must reload the integration server for the change to take effect.

    The message is written to log after coordination queue manager has been changed.

    IBM Integration Bus v10001 (IIB_FTE.IIB_FTENode) [Thread 8217] (Msg 1/1) BIP3358I: The WebSphere MQ File Transfer Edition agent ‘IIB_FTE.IIB_FTENODE’ in integration server ‘IIB_FTENode’ on integration node ‘IIB_FTE’ has been started. It is registered with the coordinating queue manager COQM and will use the configuration stored in /var/mqsi/components/IIB_FTE/4ae3cc1b-af7e-401e-9cfe-6d05bbbd4192/config/WMQFTE/config

    By issuing mqsireportproperties command coordination queue manager can be found.

     3) mqsireportproperties IIB_FTE -e IIB_FTENode -o FTEAgent -n coordinationQMgr

     Output displayed:

    COQM

     The System configurations required for the file transfers between IBM MQFT and IBM Integration Bus is now complete.

    The below Figure 3, shows that the Agents created are now in a Status: STOPPED

     

    Figure-3

    Reason: The IBM MQFT Agent AGQM has not started with the fteStartAgent command.
                    The Integration node IIB_FTE is in stopped state. Hence IIB_FTE.IIB_FTENODE agent is stopped.

  15. Starting Both the Agents

    We will now look at the steps required to Start Both the Agents that are in Stopped State

     1) Start the IBM MQFT Agent:

    ./fteStartAgent -p COQM AGQM

     Output displayed:

    BFGAG0058I: The agent has successfully initialized.

    BFGAG0059I: The agent has been successfully started.

     
    2) Start the Integration node

    mqsistart IIB_FTE

     Output displayed:

    BIP8096I: Successful command initiation

    Both the agents ready for file transfer:

    Figure 4: The Agents are both in Ready State Now

    Figure-4

  16. Test Connectivity for Both the Agents

    We will now use the MQ Explorer to test the connectivity between the two Agents we have created.

    1) Test Connectivity to the IBM MQFT Agent AGQM

    Figure 5: Test Connectivity to Agent AGQM

    Figure-5

    2) Test Connectivity to the Integration Node Agent IIB_FTE

    We now clearly see the responses received within 0.226 seconds.

    Figure 6: Test Connectivity to Integration Node Agent: IIB_FTE

    Figure-6

     

    Notes:  If both the agents receive success response, then setup is completed without any problems. We are good to go with file transfer. To address to some of the unforeseen errors that we could encounter during the setup, we have simulated a scenario where the Agents do not receive any responses are discussed in the below section, for this we have Stopped the Sender channel connected to CDQM at IIB_FTE Queue Manager.

    Figure 7: Agent “IIB_FTE.IIB_FTENODE” has not received response

    Figure-7

    We now see that the agent “IIB_FTE.IIB_FTENODE” has not received any response.

    Figure 8: Agent “IIB_FTE.IIB_FTENODE” has not received response after 5 Seconds

    Figure-8

    Note: To resolve any such issues where there was no response, you must check if all the Channels connected to the Queue Managers are in Running State

     

  17. Send Files to an Existing IBM MQ Advanced Managed File Transfer Network from an IBM Integration Bus Network

    The MQInput node receive messages from applications and connects to the Integration node by using the IBM MQ Enterprise Transport. The MQInput node transform the messages to FTEOutput Node in the form file. The file will be placed at /tmp/MQFTE_Output/ location. From this location, Managed File Transfer Agent pick the files and process further.

    A sample message flows were developed by using the IBM Integration Bus Toolkit.

    Figure 9: Message Transformation from MQInput Node to FTEOutput Node

    Figure-9

    1) To configure the MQ Input Node Properties

     On the Basic tab, for Queue name, Enter MQInput.QL

    Figure 10: MQ Input Node Properties

    Figure-10

    2) Update the FTE Output Node Properties:

    On the Basic tab, Enter the below details.

     Agent: AGQM

    Queue manager: AGQM

    File directory: /tmp/MQFTE_Output/

    File name: sample.txt

    Figure 11: FTE Output Node Properties

    Figure-11

    Notes:

    Source Agent: IIB_FTE.IIB_FTENODE
    Destination Agent: AGQM

     

  18. Testing the File Transfer from IBM Integration Bus to Managed File Transfer Agent

    The Applications put the messages at MQInput.QL, then Source Agent(IIB_FTE.IIB_FTENODE) process the messages in the file format to /tmp/MQFTE_Output/ directory.
    Directory /tmp/MQFTE_Output/ must be created at the source side where IBM Integration Bus agent is running. A Sample is provided below for reference

    bash -4.2$ ls -ld /tmp/MQFTE_Output
    drwxr-xr-x. 2 mqm mqm 40 Apr 20 09:28 /tmp/MQFTE_Output
    Then AGQM agent picks the file and process it to /tmp/MQFT_Output/ directory by using Monitor AGENT MONITOR1.

    Figure 12: Monitor Agents

    Figure-12

    Directory /tmp/MQFT_Output/ must be created at the destination side where MQ Queue Manager agent is running.
    bash -4.2$ ls -ld /tmp/MQFT_Output
    drwxr-xr-x. 2 mqm mqm 40 Apr 20 09:35 /tmp/MQFT_Output
    The message has been placed at MQInput.QL at IIB_FTE Queue Manager by the Application.

    Figure 13: Put Text Message

     Figure-13

    File has been transferred to MQFTE_Output directory and this directory is associated with FTEOutput node.
    As we configured the monitoring for this directory, file has been moved to MQFT_Output directory which is associated with Managed File Transfer Agent.

    drwxr-xr-x. 2 mqm mqm 60 Apr 21 07:57 /tmp/MQFT_Output
    drwxr-xr-x. 2 mqm mqm 40 Apr 21 07:57 /tmp/MQFTE_Output

    Notes: You can see the same time stamp for both the directories. It means file has been moved to MQFTE_Output directory at 07:57, then immediately moved to MQFT_Output.

    -bash-4.2$ ls -lrt
    total 4
    -rw-r—r–.1 mqm mqm 69 Apr 21 07:57 sample.txt
    -bash-4.2$ pwd
    /tmp/MQFT_Output
    -bash-4.2$ more sample.txt
    This message is intended to test file transfer from IIB to IBM MQ FT.
    -bash-4.2$ date
    Fri Apr 21 07:58:25 EDT 2017

  19. Transfer Log

    The Transfer log details show the completion status for all the servers and their respective destinations.

    Figure 14: Transfer Log Detail

    Figure-14

  20. Receive Files from an Existing IBM MQ Advanced Managed File Transfer Network to an IBM Integration Bus Network

    The Application places a file at /tmp/MQFT_Input directory and this directory is associated with AGQM agent, then AGQM agent transfer the file to /tmp/MQFTE_Input directory and this directory is associated with FTE Input node. Then FTEInput node process the file to MQOutput queue called MQOuput.QL in the form of messages.

    Figure 15: Message Transformation from FTEOutput Node to MQOutput Node

    Figure-15

     1) To configure the FTE Input node Properties
    On the Basic tab, Enter the below details.

    Figure 16: FTE Input Node Properties

    Figure-16

    Directory filter: /tmp/MQFTE_Input
    Filename filter*: sample.txt

    2) To configure the MQ Output Node Properties
    On the Basic tab, Enter the below details.

    Figure 17: MQ Output Node Properties

    Figure-17

     

  21. Testing the file transfer from Managed File Transfer Agent to IBM Integration Bus Agent

    bash-4.2$ ls -ld /tmp/MQFT_Input
    drwxr-xr-x. 2 mqm mqm 40 Apr 21 04:32 /tmp/MQFT_Input/
    Directory /tmp/MQFT_Input must be created at the source side where AGQM agent is running.
    Then AGQM agent process the file to /tmp/MQFTE_Input/ directory by using Monitor AGENT MONITOR2.

    Figure 18: Monitor Agents

    Figure-18

    Directory /tmp/MQFTE_Input/ must to be created at the destination side where IIB_FTE.IIB_FTENODE is running.
    -bash -4.2$ ls -ld /tmp/MQFTE_Input
    drwxr-xr-x. 2 mqm mqm 80 Apr 21 04:32 /tmp/MQFTE_Input/

    We placed the file at /tmp/MQFT_Input directory and agent AGQM processed the file to IIB Agent.
    -bash-4.2$ vi sample.txt
    This file is intended to check the transfer between AGQM to IIB_FTE.IIB_FTENODE agent
    -bash-4.2$ date
    Fri Apr 21 09:22:46 EDT 2017
    -bash-4.2$ ls -lrt
    total 4
    -rw-r–r–.1 mqm mqm 86 Apr 21 09:22 sample.txt

    The file has been moved to /tmp/MQFTE_Input directory. From there FTEInput Node processes it to MQOuput Queue called MQOuput.QL
    -bash-4.2$ ls -lrt
    total 4
    -rw-rw—-.1 mqm mqm 86 Apr 21 09:23 20170421_132318_859747_sample.txt
    -bash-4.2$ pwd
    /tmp/MQFTE_Input
    -bash-4.2$ date
    Fri Apr 21 09:23:46 EDT 2017
    -bash-4.2$ more 20170421_132318_859747_sample.txt
    This file is intended to check the transfer between AGQM to IIB_FTE.IIB_FTENODE agent

    Figure 19: Message Browser

    Figure-19

     

  22. Transfer Log

    The Transfer log details show the completion status for all the servers and their respective destinations.

    Figure 20: Transfer Log Detail

    Figure-20

  23. Errors Encountered

    Error 1: BFGCL0003E
    A message problem prevented the command from completing successfully, for queue SYSTEM.FTE.COMMAND.IIB_FTE.IIB_FTENODE on queue manager IIB_FTE. The WebSphere MQ code was 2, and the reason code was 2087.
    Solution: The problem has been resolved after created the channels between IIB_FTE (Integration Node Agent Queue Manager to Command Queue Manager (COQM).

    Error 2: BFGEL0013E
    The agent has received a reason code of ‘2087’ from WebSphere MQ while sending a message to destination queue.
    Solution: We must make sure that correct message channels should be created between all the Queue Managers. Please refer the “Set up the following message channels between your queue managers” mentioned above in this article.

  24. Conclusion

    This Tutorial shows the Integration of File Transfers between the IBM MQ File Transfer and IBM Integration Bus. It explains in detail with Step-by-instructions on two main scenarios:
    Send files to an existing IBM MQ Advanced Managed File Transfer network from an IBM Integration Bus network:
    Receive files from an existing IBM MQ Advanced Managed File Transfer network to an IBM Integration Bus network

    This also covers in-depth on demonstrating various procedures on creating the different Queue Managers (Coordination, Agent, and Command), MQ Channels, Listeners, Agents, IBM Integration Nodes

  25. Acknowledgement

    The author would like to Specially Thank, Ganesh Gopalakrishnan, Rajish E. Pattavalapil, Arundeep B. Veerabhadraiah and Premshanth Koshy, for their assistance and valuable inputs provided during the creation of this tutorial.

  26. Resources

    IBM MQ license information
    https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.pro.doc/q126140_.htm

    Configuring the Coordination Queue Manager
    https://www.ibm.com/support/knowledgecenter/en/SSEP7X_7.0.4/com.ibm.wmqfte.doc/configuring_v7_qms.htm

    Configuring Agent Queue Managers
    https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.wmqfte.doc/configuring_agent_qms.htm

    Configuring the Command Queue Manager
    https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.wmqfte.doc/setup_command_qmr.htm

    Creating the Default IBM Integration Bus Queues on a IBM MQ Queue Manager
    https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ah09090_.htm

    Preparing the Environment for IBM MQ Advanced File Transfer Edition Nodes
    https://www.ibm.com/support/knowledgecenter/en/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bc34035_.htm#bc34035_10

    Scripts to Create Artifacts Required for IBM MQ Advanced File Transfer Edition
    https://www.ibm.com/support/knowledgecenter/en/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bc34036_.htm

    Setting the Coordination Queue Manager using mqsichangeproperties
    https://www.ibm.com/support/knowledgecenter/en/SSMKHH_9.0.0/com.ibm.etools.mft.doc/an09140_.htm

1 comment on"Integrating File transfers between IBM® MQ Advanced Managed File Transfer and IBM® Integration Bus"

Join The Discussion