Ever have to follow a complicated chain of CICS tasks through multiple regions to find out why an application is hanging? Or had to work out the origin of a problem task? If so, transaction tracking is what you need!

CICS transaction tracking shows the origin of and the relationships between tasks in an application as they flow across CICS regions in a CICSplex. Take an example of users suddenly reporting their application is hanging. You know processing is via a web service request into CICS, and that this web service makes use of default pipeline processing, so you use the Tasks view in CICS Explorer to find all the instances of the CPIH pipeline transaction:

hangingCPIHs

Look at those suspend times, what is going on? Double clicking on the oldest instance of CPIH (task 494 on region IYK3ZAB4) to obtain its attributes and filtering on suspend shows:

oldestCPIH

so you can see that the work is suspended waiting on a response over IPCONN AB03 which is connected to region IYK3ZAB3. OK, now you know to go investigating tasks running on IYK3ZAB3. But wait! No need with transaction tracking, there’s a simpler way. Right clicking this top instance of CPIH gives you the option to Search Associated tasks:

useGetAssocTasks

and this shows what tasks are associated with this instance of CPIH in the Search results tab:

newtaskassoc

So now you know that the associated mirror task on IYK3ZAB3 is task 3142 which itself routed on to mirror task 2574 on region IYK3ZAB5. By default the Search results tab shows the suspend reason for each task, so you can immediately see that it is suspended on TSMAINLM and so know to check what is using all the temporary storage on IYK3ZAB5 and address this.

Transaction tracking allowed you to find the reason for the hang in just a couple of clicks!

A closer look:

How does this work? Transaction tracking is based on the concept of every user task in a CICSplex having a point of origin e.g. a web service request or an MQ message. When work first enters the CICSplex, details of its point of origin is placed into task context information called “origin data”, part of its task association data, for the first task that is created to process it. This data, the origin data, flows with the work as it moves around the CICSplex.

Looking at the above example again, the origin data for the hanging mirror task can be obtained by right clicking on it in the Search results tab and selecting “Task Association” then “Open”:

originData

This shows that the origin of the work in CICS was indeed the task mentioned above, pipeline handler task 494 on region IYK3ZAB4 for URIMAP $649090, and the web service request came in to CICS from IP address 9.20.201.152 port 57145 for user ID ALISONB. If it was the hanging mirror that had been noticed then this information could be used to track back to the owner of the work – searching on this mirror’s associated tasks would give the same results as searching from the pipeline handler task.

What about work not initiated by web service requests? What information is available to identify their points of origin? This depends on the origin itself. Work from MQ for example makes use of adapter data fields of the origin data. Their contents depends on how the work was initiated; for work initiated by the MQ Trigger monitor as a result of a message being put onto application queue ALITRIG1 on qmgr MQD6 you’d see:

adapter_data

In CICS 5.3 we enhanced the MQ Bridge to provide similar information, but of course without any initiation queue name and the bridge queue name in adapter data 3.

Work from the CICS Sockets Listener provides the IP addresses and port numbers of the local and remote session partners (from z/OS V2R2 Communications Server) and work from CICS transaction gateway can even be configured to pass on the root and current Cross Component Trace (XCT) contexts. You can even tag the work yourself by making use of the XAPADMGR global user exit.

Summary

In summary, transaction tracking shows the origin of and the relationships between tasks in an application as they flow across CICS regions in a CICSplex. Transaction tracking information is written out to SMF in a task’s monitoring data and can be useful not just for problem determination but for audit purposes too. For more details on transaction tracking and the information that is available see here.

3 comments on"Making transaction tracking work for you"

  1. Mark Laughton August 03, 2016

    Hi

    The concept is fantastic – totally agree – just what we (CICS sysprogs) and our capacity guys want to track and aggregate multiple tasks into a single business transaction, for recharging or PD.

    Except…if you’re dynamically routing webservice transactions within a cicsplex group, over MRO as we do (rather than using a DPL model, ‘cos we’ve hundreds of different transactions for different clients / many different db2 schemes and want the AOR tran to be related to, or even to match exactly the TOR tran, rather than be a generic mirror) then this just doesn’t work, right?

    The ‘origin’ of the transaction tracking data over the request stream (RZ) component is the attach of the AOR tran, not the TOR tran. And being over MRO rather than LU6.2, you can’t even aggregate / track those components of the same business transaction by UOWID.

    I’ve yet to figure a way of tying together the TOR and ‘the rest’ of a webservice business transaction in the model I’ve described, which I don’t believe must be particularly uncommon (?).

    I’ve had an RFE open in this area for a couple of years now. And we’re a CICS/PA user, which makes that doubly frustrating as we’re unable to fully exploit transaction tracking features in the recent releases.

    Regards, Mark

    • Phil_Wakelin August 18, 2016

      Mark, a simple way to tie together the TOR and the rest of a web service business transaction is to use DPL to route the request to the AOR rather than transaction route the pipeline as this supports transaction tracking. DPL routing is also more efficient than pipeline transaction routing as the pipeline runs only once in the TOR and the MRO payload is generally smaller as just the the COMMAREA or channel will need to be transmitted. See section 4.3 in redbook http://www.redbooks.ibm.com/redbooks/pdfs/sg247144.pdf for more details on the two methods.
      Best Regards, Phil

  2. Alison Lucas August 10, 2016

    Hi Mark,

    Ah yes, request streams. As you say, transaction tracking is not currently supported when using request streams. The current processing does not ship the pipeline task’s origin data with the request, so when the request reaches the AOR a new point of origin is created for the task that runs there. I’ve had a look and like you cannot find any sensible way to tie these 2 tasks together.

    This does sound frustrating. I’ve found your requirement, it’s here: https://www.ibm.com/developerworks/rfe/execute?use_case=viewChangeRequest&CR_ID=60879 for anyone wishing to vote for it. I’ve had a look at how we could address this and it is not a small piece of work, it would take a significant effort to code and test, so the more votes this requirement gets the better – it would be good to be able to address this and add another piece to the transaction tracking story.

    Regards, Alison

Join The Discussion

Your email address will not be published. Required fields are marked *