In IBM MQ 9.1.2 distributed platforms, application balancing functionality was added for use with uniform clusters. With this feature re-connectable clients are automatically moved from a queue manager (QM) with more connections to a QM with less connections. For an overview of application rebalancing within uniform clusters see the following blogs:

Prior to IBM MQ 9.1.3, application status could only be monitored by querying connections and filtering by APPLTAG. From MQ 9.1.3 on distributed, a new MQSC command DISPLAY APSTATUS and associated PCF command MQCMD_INQUIRE_APPL_STATUS have been added to display user application status (system applications are not displayed). These commands are primarily targeted for QMs in a uniform cluster although they can be used elsewhere, such as a standalone QM or QM that is part of a regular cluster if preparing to convert into a uniform cluster for example.

This blog focuses on the MQSC command DISPLAY APSTATUS which I will refer to by it’s synonym DIS APS. There are three modes of DIS APS, each defined by the type parameters ‘appl’, ‘qmgr’ & ‘local’. Each mode provides an individual perspective of the uniform cluster as shown below.

Note: for a full breakdown of DIS APS parameters see the knowledge centre documentation.

DIS APS TYPE(APPL)

  • Gives a uniform cluster wide view of the applications connected

DIS APS TYPE(QMGR)

  • Gives a view of where the applications are connected across a uniform cluster by queue manager
  • Can also see if the remote QMs are talking to us or not

DIS APS TYPE(LOCAL)

  • Gives a view of the local instances of applications connected to this QM

With these perspectives we can ask and answer the following questions:

What is running in my uniform cluster?

  • Show summary counts of the applications instances across the whole uniform cluster

Is my application balanced correctly?

  • Show me my applications and how they are spread across the members of the uniform cluster

What’s connected here and are they movable (re-connectable)?

  • Show me the application instances on a specific QM

Why wont a particular application instance move?

  • Show instance specific details relating to balancing

We will set up a uniform cluster and deploy various configurations of re-connectable and non re-connectable applications. With each configuration we will demonstrate how DIS APS can be used to query the state of applications, application instances, connections and QMs within the uniform cluster to answer the questions above. To achieve this we will complete the following tasks:

Task 1: Create a demo test environment

1.1 Set up terminals

  • Open four terminals, we will refer to them as terminals 1 through 4.

1.2 Export environment variables

  • Run the following commands in terminal 4:

    export MQAPPLNAME="MY_APP_1" to name all subsequent applications deployed in this terminal as ‘MY_APP_1’.

    export MQCCDTURL="file:///tmp/ccdt.json" to route all subsequent applications deployed in this terminal to our CCDT file.

    export PATH=$PATH:/opt/mqm/bin:/opt/mqm/samp/bin to add MQ binaries to the PATH variable.

1.3 Set up QMs

  • In terminal 4, create & start 3 QMs: QM1, QM2 & QM3

    crtmqm QM1 && strmqm QM1

    crtmqm QM2 && strmqm QM2

    crtmqm QM3 && strmqm QM3

  • Access the QMs

    run runmqsc QM* in terminals 1, 2 & 3 replacing * with 1, 2 & 3 respectively.

1.4 Set up uniform cluster

  • Configure QM1 as a full repository, define cluster channels/queue & listen on port 1415

    In terminal 1, run:

    ALTER QMGR REPOS(MY_CLUSTER)
    DEFINE LISTENER(QM1_LS) TRPTYPE(TCP) CONTROL(QMGR) PORT(1415)
    START LISTENER(QM1_LS)
    DEFINE CHANNEL(CHANNEL.QM1) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('localhost(1415)') CLUSTER(MY_CLUSTER)
    DEFINE CHANNEL(CHANNEL.QM2) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('localhost(1416)') CLUSTER(MY_CLUSTER)
    DEFINE QLOCAL(CLUSTER_QUEUE) CLUSTER(MY_CLUSTER)
  • Configure QM2 as a full repository, define cluster channels & listen on port 1416

    In terminal 2, run:

    ALTER QMGR REPOS(MY_CLUSTER)
    DEFINE LISTENER(QM2_LS) TRPTYPE(TCP) CONTROL(QMGR) PORT(1416)
    START LISTENER(QM2_LS)
    DEFINE CHANNEL(CHANNEL.QM2) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('localhost(1416)') CLUSTER(MY_CLUSTER)
    DEFINE CHANNEL(CHANNEL.QM1) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('localhost(1415)') CLUSTER(MY_CLUSTER)
  • Configure QM3 as a partial repository, define cluster channels & listen on port 1417

    In terminal 3, run:

    DEFINE LISTENER(QM3_LS) TRPTYPE(TCP) CONTROL(QMGR) PORT(1417)
    START LISTENER(QM3_LS)
    DEFINE CHANNEL(CHANNEL.QM3) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('localhost(1417)') CLUSTER(MY_CLUSTER)
    DEFINE CHANNEL(CHANNEL.QM1) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('localhost(1415)') CLUSTER(MY_CLUSTER)
  • Add the uniform cluster tuning parameter to each QMs qm.ini file:

    In terminal 4 edit these files: /var/mqm/qmgrs/QM*/qm.ini where * is one of 1, 2 or 3 and append the following tuning parameter to each:

    TuningParameters:
      UniformClusterName=MY_CLUSTER
  • Restart all QMs and wait for the AMQ9883I: Joining Uniform Cluster message to appear in each QMs AMQERR log.

    The log files can be found here: /var/mqm/qmgrs/QM*/errors/AMQERR01.LOG where * is one of 1, 2 or 3.

  • Add a SVRCONN to each QM:

    In terminals 1 through 3, run:

    DEFINE CHANNEL(CLUSTER_SVRCONN1) CHLTYPE(SVRCONN) TRPTYPE(TCP)
    SET CHLAUTH(CLUSTER_SVRCONN1) TYPE(ADDRESSMAP) ADDRESS(*) USERSRC(CHANNEL)
    SET CHLAUTH(CLUSTER_SVRCONN1) TYPE(BLOCKUSER) USERLIST('nobody')
    ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)
    REFRESH SECURITY

    Note: the security rules above are applicable when the user running the apps is a member of the mqm group, ie ‘a privileged user’. Security concepts are outside the scope of this blog. For alternative methods of defining access & authority rules see Getting going without turning off IBM MQ Security and consult the knowledge centre documentation.

1.5 Create JSON CCDT

  • Create a file of name ‘ccdt.json’ and paste in this . For this demo we will place this file in the /tmp directory, although in practice this file can be located anywhere.

    Note: we have defined each CLNTCONN twice per QM, first as a member of a QM Group and second as an independent connection to its respective QM. The former definition allows re-connectable application instances to rebalance across all QMs in the uniform cluster, while the latter is used to couple application instances to specific QMs. We will take advantage of the latter to set the uniform cluster into required states so that we have the ability to demonstrate DIS APS commands.

    Note: to enable initial load balancing for the QM group connections, connectionManagement objects are defined with clientWeight & affinity attributes set to 1 & none respectively.

We now have our environment set up! Let’s see how we can use DIS APS to inspect our uniform cluster…

Task 2: What is running in my uniform cluster?

2.1 Deploy re-connectable applications

  • In terminal 4, repeat the following command 9 times: amqsghac SYSTEM.DEFAULT.LOCAL.QUEUE QM1 > /dev/null 2>&1 &

    Note: as the get application blocks the terminal while sending reconnect status to stdout, we redirect stdout & stderr to null > /dev/null 2>&1 and fork the process in the background & so we can continue working in the same terminal.

    Note: sample program amqsghac is a get application with the re-connectable flag set.

    Note: as we have deployed all application instances to QM1, the uniform cluster will initially be in an unbalanced state.

2.2 Monitor application rebalancing

  • In any terminal 1 through 3, run the command DIS APS(MY_APP_1) TYPE(APPL) and observe the output, initially you will see BALANCED(NO):

       APPLNAME(MY_APP_1)                      CLUSTER(MY_CLUSTER)
       COUNT(9)                                MOVCOUNT(9)
       BALANCED(NO)

    Note: DIS APS with TYPE(APPL) & TYPE(QMGR) parameters can be run from any QM in the uniform cluster as they offer a cluster-wide perspective, whereas TYPE(LOCAL) offers a perspective of the specific QM targeted.

  • In any terminal 1 through 3, run the command DIS APS(MY_APP_1) TYPE(QMGR) and observe the output.

    Initially you will see BALSTATE(LOW) & COUNT(0) on QMs 2 & 3, while on QM1 you will see BALSTATE(HIGH) & COUNT(9):

       APPLNAME(MY_APP_1)                      ACTIVE(YES)
       COUNT(9)                                MOVCOUNT(9)
       BALSTATE(HIGH)                          LMSGDATE(2019-07-20)
       LMSGTIME(23:52:07)                      QMNAME(QM1)
       QMID(QM1_2019-07-19_10.19.43)
    
       APPLNAME(MY_APP_1)                      ACTIVE(YES)
       COUNT(0)                                MOVCOUNT(0)
       BALSTATE(LOW)                           LMSGDATE(2019-07-20)
       LMSGTIME(23:51:16)                      QMNAME(QM2)
       QMID(QM2_2019-07-19_10.19.56) 
    
       APPLNAME(MY_APP_1)                      ACTIVE(YES)
       COUNT(0)                                MOVCOUNT(0)
       BALSTATE(LOW)                           LMSGDATE(2019-07-20)
       LMSGTIME(23:52:06)                      QMNAME(QM3)
       QMID(QM3_2019-07-19_10.20.25)

    After a few minutes when all application instances have rebalanced, repeat the command and you will see BALSTATE(OK) and COUNT(3) on all QMs.

  • In any terminal 1 through 3, run again the command DIS APS(MY_APP_1) TYPE(APPL) and observe the output, you will now see BALANCED(YES).

    Note: BALANCED in APPL context is the state of the application in the uniform cluster (YES/NO), while BALSTATE in QMGR context is the state of application instances per QM (lOW/HIGH/OK).

    Note: the rebalance latency is governed by the rate of requests to transfer applications which is currently one request per minute

    Note: information transferred between QMs currently occurs every 10 seconds, as a result you may have to wait up to 20 seconds for correct consensus on state across the QMs.

2.3 Deploy another unique set of named application instances

  • In terminal 4, run export MQAPPLNAME="MY_APP_2" To name all subsequent applications deployed in this terminal as ‘MY_APP_2’.

  • In terminal 4, repeat the following command 6 times: amqsghac SYSTEM.DEFAULT.LOCAL.QUEUE *QMGROUP > /dev/null 2>&1 &

    Note: as we have used the QM group client connections with load balancing attributes defined, the time to realise balanced state will be greatly reduced, if not instantaneous.

2.4 Show summary counts of application instances across the whole uniform cluster

  • In terminals 1 through 3, run the command DIS APS(*Y_APP_*) TYPE(APPL) and observe the output. Each QM will report the same global perspective. You will now see two entries, one for ‘MY_APP_1’, the other for ‘MY_APP_2’:

       APPLNAME(MY_APP_2)                      CLUSTER(MY_CLUSTER)
       COUNT(6)                               MOVCOUNT(6)
       BALANCED(YES)
    
       APPLNAME(MY_APP_1)                      CLUSTER(MY_CLUSTER)
       COUNT(9)                                MOVCOUNT(9)
       BALANCED(YES)

    Note: we have used multiple wildcards to filter application name.

  • In any terminal 1 through 3, run the command DIS APS(*APP_2) TYPE(APPL) and observe the output, you will now see one filtered entry for ‘MY_APP_2’:

       APPLNAME(MY_APP_2)                      CLUSTER(MY_CLUSTER)
       COUNT(6)                               MOVCOUNT(6)
       BALANCED(YES)

2.5 Filter all applications on a specific cluster

  • In any terminal 1 through 3, run the command DIS APS(*) TYPE(APPL) WHERE(CLUSTER EQ MY_CLUSTER) and observe the output, you will see our applications as expected:

       APPLNAME(MY_APP_2)                      CLUSTER(MY_CLUSTER)
    
       APPLNAME(MY_APP_1)                      CLUSTER(MY_CLUSTER)

    Note: a partial repository QM that is a member of a uniform cluster can also be a member of another regular cluster, but not another uniform cluster. Had we set up intersecting clusters we could filter applications that do not belong to our uniform cluster, but that is outside the scope of this demo.

Task 3: Is my application balanced correctly?

3.1 Filter unbalanced applications

  • In any terminal 1 through 3, run the command DIS APS(*) TYPE(APPL) WHERE(BALANCED EQ NO) and observe the following output from which we can infer that all applications are balanced:

    AMQ8933I: Application status not found.

    Note: we have used a single wildcard to filter all application names.

3.2 Force ‘MY_APP_2’ into an unbalanced state

  • In terminal 4, repeat the following command 6 times: amqstrgc SYSTEM.DEFAULT.LOCAL.QUEUE QM1 > /dev/null 2>&1 &

    Note: sample application amqstrgc does not have the re-connectable flag set and has unlimited wait time.

    Note: environment variable MQAPPLNAME was previously set to ‘MY_APP_2’, so these non re-connectable instances will have the same name as our re-connectable instances we deployed in step 2.3.

  • As the uniform cluster attempts to rebalance ‘MY_APP_2’, the 6 original re-connectable application instances will balance themselves across QM2 and QM3 while the 6 non re-connectable application instances we just deployed will sit on QM1, as a result when rebalancing has completed our application remains unbalanced. You can monitor this process by running the command DIS APS(*APP_2) TYPE(QMGR) in any terminal 1 through 3.

3.3 Filter unbalanced applications

  • In any terminal 1 through 3, run the command from step 3.1, the output now displays ‘MY_APP_2’ as unbalanced:

    APPLNAME(MY_APP_2)  BALANCED(NO)

Task 4: What’s connected here and are they movable (re-connectable)?

4.1 Investigate

Now if we were to forget that we had forced an unbalanced state as in step 3.2, and wanted to investigate why we had found an unbalanced application as in step 3.3, we could do the following:

  • In any terminal 1 through 3, run the command DIS APS(MY_APP_2) TYPE(QMGR) and observe the output.

       APPLNAME(MY_APP_2)                      ACTIVE(YES)
       COUNT(6)                                MOVCOUNT(0)
       BALSTATE(HIGH)                          LMSGDATE(2019-07-20)
       LMSGTIME(22:49:41)                      QMNAME(QM1)
       QMID(QM1_2019-07-19_10.19.43)
    
       APPLNAME(MY_APP_2)                      ACTIVE(YES)
       COUNT(3)                                MOVCOUNT(3)
       BALSTATE(LOW)                           LMSGDATE(2019-07-20)
       LMSGTIME(22:49:04)                      QMNAME(QM3)
       QMID(QM3_2019-07-19_10.20.25)
    
       APPLNAME(MY_APP_2)                      ACTIVE(YES)
       COUNT(3)                                MOVCOUNT(3)
       BALSTATE(LOW)                           LMSGDATE(2019-07-20)
       LMSGTIME(22:49:15)                      QMNAME(QM2)
       QMID(QM2_2019-07-19_10.19.56)
  • As QM1 has properties MOVCOUNT(0) & COUNT(6), we may ask ourselves why these 6 application instances have not rebalanced…

Task 5: Why wont a particular application instance move?

5.1 Show me the application instances on a specific QM

  • In terminal 1, run the command DIS APS(MY_APP_2) TYPE(LOCAL) and observe the output:

       APPLNAME(MY_APP_2)
       CONNTAG(MQCTA39A315D01841821QM1_2019-07-19_10.19.43MY_APP_2)
       CONNS(1)                                IMMREASN(NOTRECONN)
       IMMCOUNT(0)                             IMMDATE( )
       IMMTIME( )                              MOVABLE(NO)
    
       APPLNAME(MY_APP_2)
       CONNTAG(MQCTA39A315D01821821QM1_2019-07-19_10.19.43MY_APP_2)
       CONNS(1)                                IMMREASN(NOTRECONN)
       IMMCOUNT(0)                             IMMDATE( )
       IMMTIME( )                              MOVABLE(NO)
    
       APPLNAME(MY_APP_2)
       CONNTAG(MQCTA39A315D01811821QM1_2019-07-19_10.19.43MY_APP_2)
       CONNS(1)                                IMMREASN(NOTRECONN)
       IMMCOUNT(0)                             IMMDATE( )
       IMMTIME( )                              MOVABLE(NO)
    
       APPLNAME(MY_APP_2)
       CONNTAG(MQCTA39A315D01801821QM1_2019-07-19_10.19.43MY_APP_2)
       CONNS(1)                                IMMREASN(NOTRECONN)
       IMMCOUNT(0)                             IMMDATE( )
       IMMTIME( )                              MOVABLE(NO)
    
       APPLNAME(MY_APP_2)
       CONNTAG(MQCTA39A315D01831821QM1_2019-07-19_10.19.43MY_APP_2)
       CONNS(1)                                IMMREASN(NOTRECONN)
       IMMCOUNT(0)                             IMMDATE( )
       IMMTIME( )                              MOVABLE(NO)
    
       APPLNAME(MY_APP_2)
       CONNTAG(MQCTA39A315D01851821QM1_2019-07-19_10.19.43MY_APP_2)
       CONNS(1)                                IMMREASN(NOTRECONN)
       IMMCOUNT(0)                             IMMDATE( )
       IMMTIME( )                              MOVABLE(NO)

5.2 Show instance specific details relating to balancing

  • From the output of 5.1 We can see that all ‘MY_APP_2’ application instances on QM1 have the values MOVABLE(NO) & IMMREASN(NOTRECONN), these values inform us that these instances are considered un-moveable because they are non re-connectable.

    Note: you can use the CONNTAG field as a filter to the MQSC DISPLAY CONN command to see individual QM connections from that instance, which can be useful to determine the root cause of why an application instance is considered non re-connectable. For an example see Monitoring application balancing.

Task 6: Is my uniform cluster healthy?

  • In terminal 4, run the command endmqm -i QM1 to simulate an unplanned outage and wait a couple of minutes while the uniform cluster rebalances and achieves consensus.

  • In terminal 2 or 3, run the command DIS APS(MY_APP*) TYPE(QMGR) and observe the result:

       APPLNAME(MY_APP_2)                      ACTIVE(YES)
       COUNT(3)                                MOVCOUNT(3)
       BALSTATE(OK)                            LMSGDATE(2019-08-01)
       LMSGTIME(08:22:55)                      QMNAME(QM2)
       QMID(QM2_2019-08-01_07.57.33)
    
       APPLNAME(MY_APP_2)                      ACTIVE(NO)
       COUNT(0)                                MOVCOUNT(0)
       BALSTATE(NOTAPPLIC)                     LMSGDATE(2019-08-01)
       LMSGTIME(08:17:23)                      QMNAME(QM1)
       QMID(QM1_2019-08-01_07.57.30)
    
       APPLNAME(MY_APP_2)                      ACTIVE(YES)
       COUNT(3)                                MOVCOUNT(3)
       BALSTATE(OK)                            LMSGDATE(2019-08-01)
       LMSGTIME(08:22:24)                      QMNAME(QM3)
       QMID(QM3_2019-08-01_07.57.36)
    
       APPLNAME(MY_APP_1)                      ACTIVE(YES)
       COUNT(3)                                MOVCOUNT(3)
       BALSTATE(OK)                            LMSGDATE(2019-08-01)
       LMSGTIME(08:22:55)                      QMNAME(QM2)
       QMID(QM2_2019-08-01_07.57.33)
    
       APPLNAME(MY_APP_1)                      ACTIVE(NO)
       COUNT(0)                                MOVCOUNT(0)
       BALSTATE(NOTAPPLIC)                     LMSGDATE(2019-08-01)
       LMSGTIME(08:17:23)                      QMNAME(QM1)
       QMID(QM1_2019-08-01_07.57.30)
    
       APPLNAME(MY_APP_1)                      ACTIVE(YES)
       COUNT(3)                                MOVCOUNT(3)
       BALSTATE(OK)                            LMSGDATE(2019-08-01)
       LMSGTIME(08:22:24)                      QMNAME(QM3)
       QMID(QM3_2019-08-01_07.57.36)

    QM1 has the property ACTIVE(NO), this tells us that there is a problem with QM1 and we should investigate.

    Running endmqm with the -i flag has forced an immediate shutdown of QM1, as a consequence all application instances on QM1, both re-connectable and non re-connectable have been lost.

    This can be verified by running the command DIS APS(MY_APP*) TYPE(APPL) on terminal 2 or 3 and observing the output:

       APPLNAME(MY_APP_2)                      CLUSTER(MY_CLUSTER)
       COUNT(6)                                MOVCOUNT(6)
       BALANCED(YES)
    
       APPLNAME(MY_APP_1)                      CLUSTER(MY_CLUSTER)
       COUNT(6)                                MOVCOUNT(6)
       BALANCED(YES)

    You can see that we have lost the 3 of the 9 re-connectable instances of ‘MY_APP_1’ and the 6 of the 12 non re-connectable MY_APP_2 instances, leaving us with 6 instances of each application.

  • If we had run the command endmqm with the -r flag to simulate a planned outage of QM1, the re-connectable ‘MY_APP_1’ instances on QM1 would have rebalanced across QM2 & QM3, wheras the non re-connectable ‘MY_APP_2’ instances would be lost. We will leave this as an exercise for the reader!

    Note: uniform clusters can be configured to implement high availability, although this is outside of the scope of this demo.

Summary

During this exercise, we have used DIS APS to:

  • Query application state

    • use filters and wildcards to isolate clusters, applications and application instances
    • discover an unbalanced application
    • determine which application instances are not moving
    • determine where these application instances are located and the application connection states
    • determine the root cause of the imbalance!
  • Query uniform cluster state

    • monitor the process of application instances rebalancing
    • discover unhealthy cluster nodes
  • Closing

    We’ve now gone through the basic functionality of using DIS APS, demonstrating the ability to monitor and manage your connected application instances. Now that you have a template installation, please feel free to reference the documentation and experiment with DIS APS in more detail. Also see Application balancing trouble shooting for a list of symptoms and solutions associated with application balancing.

    As uniform clusters, application balancing and DIS APS features are works in progress, the algorithm for rebalancing and inter-cluster communication timing windows are subject to change, as a consequence the behaviour of DIS APS may vary over time. Look out for upcoming blogs on further developments, and please post any thoughts on improvements or additional features that you would like to see.

Join The Discussion

Your email address will not be published.