MQ for z/OS allows messages to be put onto shared queues. These shared queues have entries in list structures in the z/OS Coupling Facility (CF). Many queue managers in the same sysplex can access those messages using the CF list structure.

Messages put on a shared queue are referenced in a Coupling Facility list structure until they are retrieved by an MQGET. Coupling Facility operations are used to:

  • Search for the next retrievable message
  • Lock uncommitted messages on shared queues
  • Notify interested queue managers about the arrival of committed messages

z/OS v2r3 introduced 2 new CFRM policy attributes:

  • LISTNOTIFYDELAY
  • KEYRNOTIFYDELAY

Prior to z/OS v2r3, there was also the SUBNOTIFYDELAY CFRM policy attribute.

We have found no benefit when testing MQ with either SUBNOTIFYDELAY nor LISTNOTIFYDELAY, so this blog will concentrate on the impact of KEYRNOTIFYDELAY.

What is KEYRNOTIFYDELAY?

The CFRM policy attribute KEYRNOTIFYDELAY allows the user to specify in their policy definitions the amount of time between notification of a system-selected keyrange monitoring exploiter instance and the notification of the other exploiter instances. The exploiter instance that receives the initial notification is selected in a round-robin fashion.

The value specified refers to the delay between the time when a single registered keyrange monitor is notified of a keyrange state transition from the empty state to the not-empty state, and the time when the other registered monitoring instances of a keyrange are notified. It can be that the other instances are never notified depending on the processing done by the initial notified monitor.

The value specified for KEYRNOTIFYDELAY can be 1 to 7 decimal digits in a range 0 to 1,000,000 microseconds.

The structure criteria for the KEYRNOTIFYDELAY parameter are:

  • The structure must be allocated as a keyed list structure.
  • The structure must be allocated in a CFLEVEL 22 or higher CF.

How do I know what values are set in my structures?

The MVS command “DISPLAY XCF,STR,STRNAME=” reports the values set, for example:

STRNAME: PERFAPPLICATION1
 STATUS: ALLOCATED
 EVENT MANAGEMENT: POLICY-BASED
 TYPE: SERIALIZED LIST
 POLICY INFORMATION:
...
  SUBNOTIFYDELAY: 5000
  LISTNOTIFYDELAY: 0
  KEYRNOTIFYDELAY: 0
 ACTIVE STRUCTURE
...  
ACTUAL SUBNOTIFYDELAY: 5000
ACTUAL LISTNOTIFYDELAY: 0
ACTUAL KEYRNOTIFYDELAY: 0
..

Note: ACTUAL values are the actual values, expressed in microseconds, for the structure as returned from the CF where the structure is allocated. In most cases the actual and the policy [LIST|KEYR] NOTIFYDELAY value will be the same. They may differ if the DISPLAY XCF command is issued after a policy change has processed but before the system has updated the [LIST|KEYR] NOTIFYDELAY value in the CF, or if the policy specification is higher than the CF model notification delay limit (NDL).

When the structure is duplexed, the actual [LIST|KEYR] NOTIFYDELAY value that applies to the duplexed structure is the value associated with the old (primary) structure instance. The value associated with the new (secondary) structure instance is not used to determine the structure’s list notification delay until/unless that structure instance becomes the old (primary) or simplex instance of the structure.

When might KEYRNOTIFYDELAY be useful to my workload?

There are several instances where setting KEYRNOTIFYDELAY may be beneficial to your workload.

  1. When workload is skewed to particular LPARs.
  2. When there is a low messaging rate with a large number of server tasks waiting for messages, allowing for spikes in workload.

Skewed workload distribution

Consider a sysplex where there is a Coupling Facility on a z14 and the connected z/OS LPARs are running on different machines, for example one z14, one z13 and one zEC12.

Expected behaviour would see the majority of work processed by the fastest LPAR with the shortest or fastest links to the CF, skewing the distribution away from the slower LPARs that may be located further away.

There may be a business requirement to distribute the workload evenly between the 3 LPARs – and it is here that KEYRNOTIFYDELAY may assist in distributing the workload more evenly.

The 2 configurations of workload that we have seen benefit from are:

  1. Trigger(first) applications, where the triggered application is started, processes one to many messages and then ends.
  2. A request/reply workload where the queue depth is low.

Both of these workloads may see the empty to non-empty transition occurring with great frequency.

By initially notifying only one selected monitor, the application is woken and given time to process the available message. If the application is not woken before the delay period is reached, other monitors will be notified, commencing a race to get the available message.

This delay means that if the selected monitor is able to process the message before the delay period is complete, there are no false triggers to the other monitors – which has the effect of reducing empty, or failed, MQGETS – or in some cases, an attempt to MQOPEN, MQGET and MQCLOSE an MQ shared queue. The cost of the open and close can be relatively expensive if seeing the first-open / last-close effect that is discussed in blog “MQ for z/OS – Performance report MP16 update“.

Example: TRIGGER(FIRST) on asymmetric system

In this first example, we have a single task that puts a 2KB message to a request queue and waits for a reply of similar size, before repeating.

The request queue is defined in a structure with KEYRNOTIFYDELAY. The queue is defined with TRIGTYPE(FIRST), where the associated process object invokes a CICS transaction that opens the queue, gets the message, performs minimal processing to generate a reply message which is put to the reply queue using MQPUT1, before closing the request queue and ending.

For this test we had a sysplex with the CF on a z14 and 4 z/OS server LPARs. Two of these LPARs were hosted on zEC12 and two were hosted on z13. On each of these server LPARs there was a single CICS region configured with a trigger monitor.

In the initial measurements with KEYRNOTIFYDELAY(0), the workload was distributed with the majority of the work processed on the 2 zEC12 LPARs – with a 90:10 split for zEC12:z13. This distribution of workload was not necessarily what we expected but does emphasize the need to measure in your own environment.

When setting KEYRNOTIFYDELAY to 500 or more, the distribution for this workload was 25% per LPAR.

Example: Mixed message size request/reply workload on artificially constrained system

The majority of tests detailed in this blog were run solely on a z14, with 3 z/OS LPARs connected to an internal CF by 3 ICP links.
In order to introduce some asymmetry, we typically configured the system such that one of the 3 LPARs was run with less CPU and ensured that there was only 1 remaining ICP link from the constrained LPAR to the CF.

Since we are using an internal CF, the response times may be significantly lower than a remote CF that is connected over physical cables, so the values of KEYRNOTIFYDELAY are relatively small so that we can show the impact of varying the attribute. On a geographically remote configuration it may be advisable to experiment with larger values, varying by 10,000 or even 100,000 microseconds.

This particular measurement has 3 separate sets of workloads running on LPAR 1 that are served by applications running on LPARs 2 and 3 where LPAR 3 is the constrained LPAR.

The workloads put messages to separate queues that have been defined on separate structures. All structures have the same KEYRNOTIFYDELAY setting for each measurement.

LPAR 1 runs the 3 workload types – small, medium and large – where medium and large messages are stored largely in shared message data sets (SMDS). There are 10 batch jobs for each workload type.

LPARs 2 and 3 both have 6 server tasks dedicated to each workload. The applications open the request and reply queues and then run in an MQGET-wait / MQPUT loop.

The following chart shows the distribution of work to the constrained LPAR as we vary the KEYRNOTIFYDELAY attribute.


The baseline value (KEYRNOTIFYDELAY 0) sees approximately one third or less of the workloads being processed on the constrained LPAR, i.e. small is 36%, medium is 30% and large is 24%.

With this particular configuration, the distribution of the medium and large workloads is more even when the KEYRNOTIFYDELAY is set to 250 or higher.

The small message workload however demonstrates different distribution – a value of 250 gives a reasonably even distribution between the 2 server LPARs, but higher values result in the workload skewing towards the constrained LPAR such that with a value of 1000, the constrained LPAR processes 74% of the messages, with the more powerful LPAR only processing 26% of the workload.

With a KEYRNOTIFYDELAY setting of 100,000 the skewing was 97% in favour of the constrained LPAR as the transition from empty to non-empty did not occur with any great frequency – the LPAR was unable to keep up with the work as it arrived.

The setting of the KEYRNOTIFYDELAY attribute had a clear impact on the message processing rate, particularly for the small message workload, e.g.


Whilst the medium and large message workloads (not shown) saw similar throughput regardless of KEYRNOTIFYDELAY setting for this configuration, the small message workload transaction rate dropped by up to 30% when the workload was skewed towards the constrained LPAR. In this case, it was due to the LPAR being CPU constrained.

Example: Single size request/reply workload on artificially constrained system

This example reuses the LPAR configuration in the previous measurement but rather than using 3 workloads with small, medium and large messages, instead uses 3 workloads with only small messages.

In this example we see the same distribution exhibited by all 3 sets of workloads as the KEYRNOTIFYDELAY attribute is varied. Of more interest is the impact that KEYRNOTIFYDELAY has on the overall transaction rate as demonstrated in the following chart:



In this configuration we see the most even distribution of work when the KEYRNOTIFYDELAY attribute is set to 1000 – but the peak throughput is achieved with a value of 250 or less.

With this configuration, the default behaviour was to skew the workload 80:20 in favour of the more powerful LPAR – and all 3 workloads show similar behaviour.

When configuring one structure to have KEYRNOTIFYDELAY(1000000) and the other structures to have KEYRNOTIFYDELAY(0) we see different behaviour, i.e. the skewing is:

  • 87:13 to the more powerful LPAR for the workloads running on structures with KEYRNOTIFYDELAY(0)
  • 97:3 to the less powerful LPAR for the workload running on the structure with KEYRNOTIFYDELAY(1000000).

In this example, the less powerful LPAR is constrained for CPU, and whilst the total throughput for the 3 workloads is similar to the baseline, the workload using the structure with KEYRNOTIFYDELAY(1000000) sees a reduction in throughput of 40%.

This means that enabling KEYRNOTIFYDELAY for one structure can impact the distribution of other workloads, even when they use structures without KEYRNOTIFYDELAY set.

Low messaging rate with large numbers of server tasks waiting for messages

In the configuration where you have large numbers of server tasks waiting to process intermittent periods of peak processing, the low messaging rate may cause many empty gets relative to the number of valid gets, where a valid get is one that completes with message retrieval.

APAR 99242 attempts to resolve this issue by only notifying a subset of waiting tasks on transition from empty to non-empty, however if the workload increases dramatically when the queue is in non-empty state, the currently active subset of tasks will have to process the workload, as the notification only occurs at empty to non-empty transition.

Example: Excessive get-wait tasks on asymmetric system

This example was run on the 3 LPAR system where:

  • LPAR 1 had 10 dedicated processors and 3 ICP links to the local CF
  • LPAR 2 had 3 dedicated processors and 3 ICP links to the local CF
  • LPAR 3 had 10 dedicated processors and 3 ICP links to the local CF

In this instance, there were 2 queue managers on each LPAR, each connected to a single queue sharing group.

The queue managers on LPARs 2 and 3 each have 20 long-running server tasks connected, for a total of 80 server applications.

Initially 200 clients connect to one queue manager on LPAR 1, and then put/get messages to/from shared queues that are served by the applications running on LPARs 2 and 3.

At the mid-point of the test, an additional 400 clients are connected to the second queue manager on LPAR 1, and these clients then put/get messages to/from the same shared queues.

The rate that the messages are put by the clients could be processed by a much smaller number of server applications – so there is the potential for many server tasks being woken and failing to get messages, i.e. resulting in many empty gets.

For this workload, setting KEYRNOTIFYDELAY to a non-zero value resulted in the workload distribution moving from 80:20 in LPAR 3’s favour to a 50:50 split.

In this measurement, the use of KEYRNOTIFYDELAY also reduces the number of empty gets, which in turn reduces the cost of processing the workload.

MQGETs attempted for each successful MQGET

KEYRNOTIFYDELAY 0 Greater than 0
LPAR 2 116 16
LPAR 3 33 16

In these measurements setting a non-zero value for KEYRNOTIFYDELAY gave immediate benefit in reducing the number of empty gets, i.e. reducing the number of gets attempted compared the number of successful gets as reported by the MQ Accounting Class(3) data.

The following chart shows the impact to the cost of processing each message successfully gotten as the KEYRNOTIFYDELAY value increases.

  • On LPAR 2, the cost reduces to 20% of the baseline due to the reduction in the large number of empty gets.
  • On LPAR 3, the cost reduces to 60% of the baseline as the original number of empty gets was much less.




Note: The cost when processing the messages is higher on LPAR 3 due to the higher cost of MQGET. For example when KEYRNOTIFYDELAY is 1000000:

LPAR Average cost of MQGET Gets (attempted / successful) Cost of a successful MQGET
2 1.157 15.31 17.72 uSeconds
3 2.887 16.29 47.04 uSeconds

Note: Cost an a successful MQGET is calculated by “Average cost” multiplied by “Gets Attempted / Gets Successful”.

Example: Variable workload, with excessive get-wait tasks on symmetric system

This example was run on the 3 LPAR system where:

  • LPAR 1 had 10 dedicated processors and 3 ICP links to the local CF
  • LPAR 2 had 3 dedicated processors and 3 ICP links to the local CF
  • LPAR 3 had 10 dedicated processors and 3 ICP links to the local CF

In this instance, there were 2 queue managers on each LPAR, each connected to a single queue sharing group.

The queue managers on LPARs 1 and 3 each have 40 long-running server tasks connected, for a total of 160 server applications.

Initially a low workload is driven from LPAR 2 – driving at 2 transactions per second. This typically causes many of the server applications to attempt and fail to get the message, resulting in many empty gets.
As time passes, a higher workload is run for short periods, to simulate busy periods – this is where the many server tasks are required and the ratio of empty gets should decrease.

For this particular configuration, the distribution of workload between LPARs 1 and 3 was evenly split regardless of KEYRNOTIFYDELAY setting.

The number of empty gets in the KEYRNOTIFYDELAY(0) measurement is significantly reduced when the attribute is set to non-zero values. Once the KEYRNOTIFYDELAY is set to 100000, virtually every MQGET is successful as demonstrated in the following chart:


By reducing the number of failed gets, there is a reduction in the cost of processing each message successfully gotten as the KEYRNOTIFYDELAY value increases.

  • On both LPARs 1 and 3, the cost reduces to 20% of the baseline cost.

The following chart shows the impact to the cost of processing each message successfully as well as the average transaction rate for the entire workload. This transaction rate is weighted towards the peak processing periods but shows a 34% improvement over the baseline when KEYRNOTIFYDELAY is set to 100000.



Example: Variable workload, with excessive get-wait tasks on symmetric system with PM99242 applied

This measurement is a repeat of the previous measurement, except that the service parameter is enabled as per PM99242.

The number of empty gets per successful get is significantly lower due to the APAR determining an appropriate number of get-wait tasks to notify, but with non-zero KEYRNOTIFYDELAY, the number of empty gets is still reduced i.e.


By reducing the number of empty gets, there is again a reduced cost of processing each message successfully gotten as the KEYRNOTIFYDELAY value increases:

  • On LPAR 1, the cost reduces by up to 50% of the baseline cost
  • On LPAR 3, the cost reduces by up to 42% of the baseline cost

The following chart shows the impact to the cost of processing each message as well as the average throughput achieved throughout the measurement period.


The increase in transaction cost when KEYRNOTIFYDELAY between 150 and 1000 is due to increased CF cost which manifests as sub-channel delays in Coupling Facility Subchannel RMF(TM) report. It is not immediately obvious why these measurements show delays.

How do I know what to set KEYRNOTIFYDELAY to?

Experiment – your system configuration and workload may be different from your peers.

Using a value at the higher end of the range will ensure less-favoured LPARs are given the best opportunity to process messages but this may have several side-effects including:

  • Skewing of workload to the less-favoured LPARs. If the messages are arriving faster than the slower LPAR can process, the empty to non-empty transition will occur less frequently – which can result in the messages being processed mostly by the slower LPAR.
  • Increasing the response time. If the fastest LPARs are prevented from processing messages when they are able, the slower LPARs may cause the response time to increase and transaction rate to decrease.

Beware the attribute is set at a structure level, so will apply to all queues defined in that structure.

If there are multiple queues in the same structure each processing different sized messages, what may be a good value of KEYRNOTIFYDELAY for the large message workload may not be optimum for a small message workload.

Conclusions

KEYRNOTIFYDELAY may assist in distributing workload more evenly between different levels of IBM Z hardware and can reduce the number of false trigger / empty gets, thereby reducing transaction cost.

Determining the optimal value for KEYRNOTIFYDELAY will require experimenting on your system and your particular workloads – including reviewing the size of the messages in your workloads.

If even distribution is desired, tune to the least powerful LPAR located furthest away from the CF.

Enabling KEYRNOTIFYDELAY can affect all users of the queues defined to the structure being changed.

Enabling KEYRNOTIFYDELAY on one structure can impact workload on other structures if it causes resource shortage (CPU / CF link busy).

There may be an impact to response time if more workload is processed by slower systems.

If the system processing the workload is unable to keep the queue depths low enough to drive the empty to non-empty transition, the workload can become heavily skewed to the slower system.

Join The Discussion

Your email address will not be published.