Application rebalancing across a uniform cluster

So far in this series, you have learned the basics of uniform clusters. You have also seen how applications are rebalanced across these with an IBM MQ uniform cluster tutorial.

Now you need to understand what is actually involved in the rebalancing process, and how you can investigate unexpected behaviors. To do this, you need to understand what attributes the uniform cluster monitors and how their values influence balancing.

Interrogating your uniform cluster

You can interrogate the status of the different components of your uniform cluster in different layers, which is useful for debugging if you find that the app instances hosted in the uniform cluster are not balancing as expected. The following sections examine these layers in detail using the examples found in the IBM MQ docs, to understand the attributes involved in rebalancing.

The following command allows you to query the status of all app name groups in a uniform cluster:

DIS APSTATUS(*) type(APPL)

Typical output from this command looks like this:

Here, you can see the attributes that the uniform cluster keeps track of related to a single app name group. The following list breaks down the key balancing attributes and puts them into context of this example:

• APPLNAME is the app name group for which the information is being displayed and each app name group connected to the uniform cluster has a separate listing. You can see here that MYAPP is the only value of APPLNAME shown, which indicates that it is the only app name group connected to the uniform cluster.
• CLUSTER is the name of the uniform cluster the app name group is connected to, which is UNIDEMO in this case.
• COUNT is the number of instances of the app name group connected to the uniform cluster. In this example, there are 8 instances of MYAPP connected to UNIDEMO.
• MOVCOUNT is the number of connected app instances that are eligible for rebalancing. In this example, all app instances are eligible for rebalancing since MOVCOUNT = COUNT.
• BALANCED indicates whether the instances of MYAPP are balanced across the uniform cluster. In this example, you can see that they are not balanced.

You can then look at the status of all queue managers in a uniform cluster using this command:

DIS APSTATUS(*) type(QMGR)

Typical output of this command looks like this:

This example output shows the status of each queue manager in the uniform cluster, in relation to each app name group that is connected to it.

• QMNAME is the name of the queue manager. Here, there are 3 queue managers (UNID001, UNID002, UNID003).
• APPLNAME is the app name group connected to that queue manager. In this example, the only app name group connected to the uniform cluster is MYAPP, so you only see this in the listed output.
• COUNT is the number of instances of the app name group connected to the queue manager. In this example, you can see that UNID001 has 6 app instances of MYAPP, and UNID002 and UNID003 each have 1 instance.
• MOVCOUNT is the number of app instances that are eligible for rebalancing in each queue manager. In this example, all app instances are eligible for rebalancing because MOVCOUNT = COUNT for all queue managers.
• BALSTATE shows the relative balance of the app instance indicated by APPLNAME in each queue manager and can have values HIGH, LOW, or OK. Because UNID001 has more instances of MYAPP connected to it than UNID002 and UNID003, its BALSTATE is HIGH, while the BALSTATE of the other 2 queue managers is LOW.

Now you can look at the status of each app instance within a particular queue manager using this command:

DIS APSTATUS(*) type(LOCAL)

Typical output would look like this:

In this example, the information for the app instances connected to UNID001 is listed after the uniform cluster became balanced. Looking at the key balancing attributes:

• MOVABLE shows whether the app instance is eligible for reconnection across the uniform cluster. In this example, you can see all 3 app instances in UNID001 can be reconnected.
• IMMREASN states the reason why the app instance is not reconnectable. In this example, because all the displayed app instances are reconnectable, IMMREASN is NONE.

Rebalancing scenarios

The queue managers in the uniform cluster periodically share and synchronize state to ensure the balance of application instances is consistent cluster-wide. As mentioned above, the BALSTATE attribute indicates the relative balance of a queue manager with regards to a particular app name group. The number of application instances that are available to move to achieve a balance is indicated by the MOVCOUNT attribute value for that app name group. Ideally, the value of MOVCOUNT would be equal to the value of COUNT to allow unconstrained balancing, but this is not always the case.

The following animations illustrate some typical rebalancing scenarios to help visualize how MOVCOUNT might influence balancing within a uniform cluster.

This animation shows how rebalancing occurs when all connected app instances are eligible for rebalancing. In this example, QM1 has 6 instances of app1 (causing BALSTATE : HIGH), and QM2 has 2 (causing BALSTATE : LOW). All of these app instances can be rebalanced, so QM2 requests 2 app instances of app1 from QM1 so that both queue managers have an equal number connected to them. This is an ideal scenario since all app instances could be rebalanced so there was no impediment to the uniform cluster becoming balanced.

The following animation shows a rebalancing scenario where there are some app instances that cannot be rebalanced. Here, QM1 again has 6 instances of app1 connected, but only 4 of these are movable. QM2 has only 2 instances of app1, so again requests QM1 for 2 more. 2 of the 4 movable app instances from QM1 are reconnected to QM2 and the uniform cluster is balanced. Admin intervention might be useful to determine why there are unmovable app instances, although it is not necessary since the uniform cluster was still balanced.

Here you see a scenario where the number of movable app instances is exhausted. QM1 has 15 instances of app1, of which only 3 are movable. QM2 has 5 instances of app1 so requests 5 from QM1 to become balanced. Since QM1 does not have 5 movable instances of app1 connected to it, it is only able to reconnect 3 of these to QM2. The uniform cluster is still left unbalanced. This situation would require intervention to understand why so many of the app instances of app1 connected to QM1 are not eligible for reconnection.

The following animation illustrates the scenario where there are no movable app instances. Here, QM2's BALSTATE is LOW, so it requests 5 instances of app1 from QM1. QM1 has 15 instances of app1, but none of these are eligible for rebalancing. This means that app instances cannot be reconnected to QM2, and the uniform cluster remains unbalanced. This is an extreme scenario and intervention would be necessary to ensure the uniform cluster is able to behave optimally.

Intervention steps for non-reconnectable applications

When you encounter scenarios where you have non-reconnectable application instances, you need to intervene to work out the causes for this. This can be done using the commands discussed at the start of this article, and the process for doing so is as follows:

1. Look at each app name group and see if any are unbalanced, using DIS APSTATUS(*) type(APPL) and checking if COUNT equals MOVCOUNT.
2. If there are imbalances, look at the balance of app instances in each queue manager using DIS APSTATUS(*) type(QMGR) and check if there are any queue managers where COUNT does not equal MOVCOUNT.
3. For these queue managers, look at the individual app instance they are hosting using DIS APSTATUS(*) type(LOCAL) and identify the reason they are unmovable (given by the value of IMMREASN).
4. You can then look up the values of IMMREASN in the table at the bottom of the IBM MQ application balancing troubleshooting guide to identify the action you need to take to address the issue.

Summary

In this article, you learned techniques that you can employ to understand the state of the queue managers in a uniform cluster and their connected applications. You also learned about typical rebalancing scenarios and the steps you can take to address non-reconnectable applications.

To prevent your applications from becoming non-reconnectable in the first place, ensure they fulfil the essential criteria found in the table at the end of the uniform cluster setup tutorial.