This is the second in a short series of performance blogs for MQ on z/OS to draw your attention to documents relating to performance updates. In this blog we talk about some of the updates to MP16 – the capacity planning and tuning guide for MQ on z/OS.
The latest versions of MP16 and other performance documents can now be found on the mqperf github repository.
Recently the performance team has been looking into some issues that customers have seen within the Queue Sharing Group environment.¬† Additionally there is some updates that have come from our move to z14, such as CFCC level 22 and Crypto Express6S.
In this blog I am going to mention the following 3 issues, although they are covered in more depth in MP16’s Advice section:
- First-open / last-close effect on shared queues.
- Attempting to open a queue that has not been defined.
- Using a common MSGID or CORRELID with deep shared queues.
Addressing the issues identified in these items can result in significant CPU reductions.
First-open / last-close effect on shared queues.
With shared queue, there is a first-open and last-close effect which can increase the cost of opening a queue.
When an application opens a shared queue, if no other application on the queue manager already has the queue open, then the queue manager has to go to the coupling facility to register an interest in the queue and to get information about the queue. This is the first-open effect. If the queue manager already has the queue open, the information is already available and the queue manager does not need to register.
When an application closes a shared queue, if no other application on that queue manager has the queue open, then the queue manager has to go to the coupling facility to deregister interest in the queue. This is the last-close effect.
Some applications naturally have the queue open for a long time such as channels or long running transactions. With high throughput, short-lived transactions, the queue may always have at least one application with the queue open. In these cases the first-open and last-close effects are not seen.
If you have short running transactions with throughput such that the queue is not open all of the time, this can result in each transaction experiencing first-open and last-close effects.
If the queue is being accessed by applications connected to different queue managers in the queue sharing group, the impact of first-open and last-close can be magnified further.
We created the first-open and last-close effect across 3 queue managers and the impact is shown in the chart below.
This chart shows that the time spent waiting for the MQOPEN to complete when the open is only attempted on 1 queue manager was on average 30 microseconds. When the opens were attempted on 2 queue managers concurrently, the MQOPEN time increased to 134 microseconds, and this increased further with 3 queue managers to 490 microseconds.
This increased time is largely the result of waiting for a lock and is shown in the MQ Accounting data as DMCSEGAL latch time. A less responsive CF may result in increased DMCSEGAL time.
To reduce the impact of first-open and last-close effects, use an application connected to each queue manager that opens the shared queue and then goes into a long sleep. In our measurements this reduced the open time to 2 microseconds irrespective of the number of queue managers that had applications attempting to open the queue.
Attempting to open a queue that has not been defined.
When an MQOPEN of a queue is requested, the queue manager will perform a look-up of local storage to get information about the queue. If successful, queue open processing will continue.
If the queue name is not found in local storage and the queue manager is part of a Queue Sharing Group, the queue manager will look for the queue in the DB2 table CSQ.OBJ_B_QUEUE. This check can be relatively expensive, of the order 8-10 times that of a successful queue open. There is cost incurred in both by the queue manager in its Db2 server threads and in the Db2 master address space.
Measurements on our systems attempting 1 million MQOPENs, showed an average cost of approximately 59 microseconds in the queue manager and 54 microseconds in the Db2 master address space for each attempt to open an undefined queue. In an environment where the MQOPENs are less frequent, higher costs may be observed.
The MQ Class(3) Accounting data will show information such as:
VKW1 Batch Jobname:VKW1PUT Userid:SHARKEY Start time Feb 22 07:10:24 2018 Started this interval Interval Feb 22 07:10:24 2018 - Feb 22 07:10:25 2018 : 1.042905 seconds Other reqs : Count 2 Other reqs : Avg elapsed time 201 uS Other reqs : Avg CPU 31 uS Other reqs : Total ET 0.000402 Seconds Other reqs : Total CPU 0.000062 Seconds == DB2 activity : 1 requests > Average time per DB2 request-Server : 321 uS > Average time per DB2 request-Thread : 321 uS > Maximum time per DB2 request-Server : 321 uS > Maximum time per DB2 request-Thread : 344 uS > BytesputtoDB2: 0 > Bytes read from DB2 : 0
In this example, the job “VKW1PUT” is attempting to open an undefined queue, which results in a request to DB2 taking 321 microseconds, with no data being read from DB2.
Using a common MSGID or CORRELID with deep shared queues.
MQ shared queues can contain many millions of messages and with the use of unique index keys, such as message ID, can enable the application to get specific messages from the queue with consistent response times.
However, MQ shared queues are not optimised for MQGET when the queue is deep and a common key is used to identify many messages. As the number of messages with the common key increases, the cost increases as does the time to complete the MQGET.
If it is necessary to use a common value in the index of a shared queue, best MQGET performance will be achieved when that common value applies to less than 50 messages. Where more than 6,000 messages have the same value and an MQGET using the index is specified, the cost of the MQGET may be of the order of 15 times that of a get where only 1 message matches the search criteria.
The following chart illustrates the increase in MQGET cost as the number of message sharing a common key increases.