There are many different skill levels of CICSPlex SM users. Some have been using and supporting CICSPlex SM for years, others might be brand new to CICSPlex SM. Regardless of the experience level, we often have support requests opened with CICSPlex SM Level 2 Support asking “How do I do xxxx in CICSPlex SM?” What they are trying to do can vary quite a lot, from the simple to the complex.

One question I was asked recently was: How do I monitor my CICS regions to make sure that a certain transaction id is always running, and to issue a message for the operations staff when the transaction is found to be no longer be running?

The business need to do this is obvious. But the path to accomplish it might not be quite as clear. Allow me to introduce you to the Real Time Analysis (RTA) and MAS Resource Monitoring (MRM) feature of CICSPlex SM.

For this example, suppose you have an application transaction ‘ABCD’ that you want to make sure is always running in your backend application owning regions (AORs). Your CICS topology might look like the following:

PRODPLEX CICSplex Sample Configuration
For help setting up an environment like this, see my blog entry Creating a Basic CICSPlex SM environment.

There are a three main resources in CICSPlex SM that you must define to accomplish this monitoring, plus two ‘container’ type resources:

EVALDEF (Evaluation Definition) – describes what resource you will be monitoring, the criteria to be used to determine if the condition is ‘true’, and how frequently CICSPlex SM needs to perform its checking in the CICS regions.

ACTION (Action Definition) – describes what action to take when the above Evaluation Definitions changes status from true to false, and from false to true.

RTADEF (Real Time Analysis Definition) – ties the EVALDEF and ACTION together. Also, determines how frequently CICSPlex SM looks at the status (true or false) of the EVALDEF.

RTAGROUP (RTA Group) – this is a container type group, that only serves to hold the RTADEFs you want to combine. It is still required even if there is only going to be one RTADEF in it.

RTASPEC (RTA Specification) – this is the other container type group. It points to the RTAGROUP on one side, and is associated with the target of where you want to implement the monitoring on the other. I will discuss this in more detail later.

You have a couple of choices when it comes to creating these definitions: Online through the CICSPlex SM Web User Interface (WUI), or through a batch job. I will walk you through the WUI definitions here.

To start, logon to the WUI, expand the +Administration twistie, and select RTA MAS resource monitoring:

Expanding the Admin twistie and selecting RTA MAS resource monitoring

First, create the RTA Specification. From the RTA MAS resource monitoring menu, click the Specifications link:

RTA - Select Specs

Then click the Create button. On the create screen, there are only two fields you will be filling out, and they are right at the top:

RTA specification name – An 8-character specification name of your choosing. I called mine ‘MYRTASPC’.

Description – This is strictly for your own documentation.

Once you fill these two fields, click the Yes button at the bottom of the screen to complete the action. You should now see the created RTASPEC:

RTA - RTASPEC Created

Now click the Go Back To Last Menu button at the top:

WUI Navigation - Go Back to last menu

and select the Groups option:

RTA - Select Groups

On the RTA Groups screen, click the Create button. Fill in the fields as follows:

RTA - RTAGROUP Create

RTA Group – An 8-character group name of your choosing. I called mine ‘MYRTAGRP’.

Description – This is strictly for your own documentation.

Click the Yes button to complete the create. You should see confirmation, and the new entry listed like the following:

RTA Group Created

Next we need to tie the add the RTAGROUP to the RTASPEC. With the RTAGROUP listed, put a check mark next to it and click the Add to RTA specification button:

Add RTAGROUP to RTASPEC

There is only one field to fill out, and that is the RTA specname. Type in the name of the RTA specification you created. In this example, I called it ‘MYRTASPC’. Once you fill the RTA specname in, click the Yes button to complete the action.

Now go back to the RTA MAS resource monitoring menu and select Evaluations:

RTA Select Evaluations

Once there, click the CREATE button. You will be looking at the EYUSTARTEVALDEF.CREATE view (the view or menu names that I refer to are always at the bottom right of the browser):

RTA EVALDEF Fields

The fields you need to fill out are:

Name – I chose ‘TRANGONE’ for this example. You are free to pick your own EVALDEF name.

Description – This is strictly for your own documentation.

Sample interval – This is an important one. It determines how often CICSPlex SM will send a probe down to the MAS and perform the checking requested by this EVALDEF. The default is 300 seconds, or 5 minutes. If you leave it set to that for this example, that means every 5 minutes CICSPlex SM will send a probe to the requested regions and check to see if transaction ABCD is running. If for some reason the task ended one second after CICSPlex SM checked, it would be another 4 minutes 59 seconds before CICSPlex SM came back and found it gone. When I do testing of a new setup, I usually set it down to 10 seconds so I do not have to wait so long to see results. Just remember that the lower this number, the more frequently CICSPlex SM performs checking, and the more overhead there would be.

Resource table – This example is to watch active tasks and ensure a specific transaction id is running, so I selected the TASK table. You do not have to remember all the various names of the tables. In the WUI, anytime CICSPlex SM has a list of choices for you, there will be a ‘pencil’ selector to the right of the input field. If you click the pencil, you get a list of all possible valid values. Refer to the CICS Transaction Sever for z/OS (CICS TS) documentation for descriptions of all the CICSPlex SM resource tables.

Instance identifier of evaluated resource – You need to specify something here, but you will use an asterisk (*) to say you want any instance. This field applies to the primary key of whatever Resource Table you selected above. In the case of the TASK table, the primary key is the task number. Since we do not care what task number the ABCD transaction is, we leave this set to ‘*’.

Method of evaluating results in result set – There are a number of options to choose from here (For example: All, Any, Sum, Min, and Max), but all we want to do in this example is make sure at least one ABCD transaction is running. Therefore, select ‘Cnt’ for Count.

Separate task indicator – This defaults to ‘No’, which is fine. The probe sent down to do the checking will then run under one of the existing CICSPlex SM long running tasks.

Field being evaluated – You can leave this blank. We are only checking to see if an ABCD transaction is running so this field is not applicable.

Evaluation type – Select ‘Value’.

Evaluation logical operator – Set this to ‘Eq’ for Equal.

Evaluation logical operator – set this to ‘0’ (zero.)

Severity assigned when result meets criteria – This one is entirely up to you. The values in the dropdown box range from VLS (Very low severe) to VHS (Very high severe). The default we are taking is ‘Lw’ for Low warning.

Threshold evaluation type parameters – All of these can be left blank, since we earlier indicated that we were going to use Evaluation type ‘Value’.

View that may provide extra information – Leave this field blank

Filter string – Here is where you finally get to specify the transaction code. In this field, type ‘TRANID=ABCD.’ without the quotes, but the terminating period is required. You can change ‘ABCD’ to be whatever transaction is important at your shop.

That is it. Once the above fields are filled in, click the Yes button to create the EVALDEF.

Your screen should now look like the following:

RTA EVALDEF Created

With that done, we will move on to creating the ACTION definition. Go back and select Actions:

RTA Select Actions

Click the Create button. You will be looking at the EYUSTARTACTION.CREATE view:

RTA ACTION fields

The fields you need to fill out are:

Action – I used ‘TRANGONE’ for this example. It is the same name as the EVALDEF. It can be different.

Description – This is strictly for your own documentation.

Generate event – Select ‘Yes’.

Action Priority – This defaults to ‘1’, and can range from 1 to 255. It is used to determine the sort order on the Outstanding Events view (WUI Main Menu –> Real Time Analysis (RTA) –> Outstanding events.)

External message sent when event occurs and External message sent when event is cleared – These fields give you 30 characters that can show up in the external message itself, when the Event is raised and when it is cleared. I have been as meaningful as I could in 30 characters or less in the example. You will see the result later in this blog.

Generate SNA generic alert – Select ‘No’. CICSPlex SM has the capability to generate alerts that go into Netview. If you are using NetView and want to utilize this feature, refer to Enabling a CMAS to send generic alerts to NetView in the CICS TS documentation.

The next three fields:

CMAS to which NetView attached

Message text when alert is raised

Message text when alert is cleared

we will leave blank, since we are not utilizing the NetView facility in this example.

MVS automatic restart – This defaults to ‘No’, and you probably want to leave it that way. If you chose ‘Yes’, and the event occurred, CICSPlex SM will immediately cancel the region and initiate a restart through MVS automated restart manager (ARM.) There are a number of requirements for this to occur. See Implementing MVS automatic restart management in the CICS TS documentation for details.

Click the Yes button to create the ACTION definition.

Your screen should now look like the following:

RTA ACTION Created

With that done, we will move on to creating the RTADEF definition. Click the Go Back To Last Menu button at the top, then click the Definitions link:

RTA Select Definitions

Click the Create button. You will be looking at the EYUSTARTRTADEF.CREATE view:

RTADEF Fields

The fields you need to fill out are:

Name – I am using ‘TRANGONE’ again in this example. It is the same name as the EVALDEF and ACTION definitions, but it can be different.

Description – This is strictly for your own documentation.

Execute evaluation modification string – Select ‘No’. CICSPlex SM does have the capability to take action, such as purging a task, but for this example, I’ve limited the scope to simply putting out a message.

Analysis interval – The default here is 300 seconds, or 5 minutes. This, like when we were creating the EVALDEF and had field Sample interval which says how frequently CICSPlex SM is going to perform checking. What it is going to check in an RTADEF is the status (true or false) of the EVALDEF pointed to in the next field. Pay close attention to these polling time fields (Analysis interval and Sample interval.) They interact and work with each other, and need to make sense in that context. For example, it would make no sense to have the EVALDEF Sample interval set to 300 seconds and the RTADEF Analysis interval set to 10 seconds. That would mean every 10 seconds CICSPlex SM would be checking the state (true or false) of the EVALDEF, when it only had the possibility to change once every 300 seconds. These two polling time fields should usually be kept the same. For this example, I have chosen 10 seconds for each, to simplify and speed up my testing. You should choose values that will alert you of a condition in sufficient time, but minimize the overhead of polling and checking too frequently.

Action definition name – Type ‘TRANGONE’ here, or whatever name you chose for the ACTION definition you created in the last step.

COUNT fields – By default, these are all set to ‘1’. It is probably a good start, but be aware that you can change them if needed. For example, if your Analysis interval was 10 seconds, and Count of true evaluations before LW raised was set to 3, it would take 3 consecutive 10-second Analysis intervals of finding the condition in the EVALDEF “true” before an alert was raised. This might be useful if you wanted to monitor a task that runs continuously, but periodically it terminates itself and restarts a few seconds later. A 10-20 second gap might be acceptable, but if it goes on longer than that, you would want to raise an alert. Alternatively, you could simply change the Analysis interval to take that into account. There are a lot of options here.

Evaluation expression – Type ‘TRANGONE’, or whatever name you chose for the EVALDEF you created in the first step above.

Click the Yes button to create the RTADEF definition.

Your screen should now look like the following:

RTA RTADEF Created

Finally, we add the RTADEF to the RTAGROUP we created earlier. Put a check mark next to the RTADEF, and click the Add to RTA Group button:

RTA Add RTADEF to RTAGROUP

On the resulting view, type the RTA group name you created earlier:

RTA Add RTADEF to RTAGROUP

Click the Yes button to complete the action.

With the resources defined, you might want to see a graphical representation of how they are tied together. That is where the WUI MAP feature can help. Return to the RTA Specifications screen and put a check in the box next to the MYRTASPC RTA specification and then click the Map button:

RTA MAP feature

The resulting display looks like this:

RTA MAP Feature Details

This was a simple scenario, so the display in this case simply allows us to verify that the pieces are tied together properly. If you use other parts of CICSPlex SM, such as Workload Manager (WLM), you can use the Map feature on those definitions as well. Navigating through the various Map displays would be something I could showcase in a future blog entry. Let me know through the Comments section below if you would be interested in something like that.

Now we get to testing. To do that, we will install the RTASPEC. On the RTA Specifications screen, put a check in the box next to the RTASPEC MYRTASPC, then click the Associate CICS System button:

RTA Associate RTASPEC to MAS

You will be looking at the EYUSTARTRTASPEC.ADDSYSDEF view:

RTA Associate RTASPEC to MAS 2

Fill in the fields as follows:

CICS system – For this example, I have typed ‘AOR001’ as the single region name where I want to associate this RTASPEC. You could have also associated the RTA specification with a group of CICS regions by selecting the Associate CICS Group button instead of Associate CICS System on the RTA Specifications screen.

Click the Yes button to complete the association. You should receive the following confirmation:

RTA Associate RTASPEC to MAS 3

At this point, everything is done. AOR001 was not running so I started it up. Then I went to the EYULOG for the CMAS001 where AOR001 connected, and I see confirmation that my RTASPEC was installed:

EYUCL0012I CMAS001 Connection of CMAS001 to AOR001 complete.
EYUTS0003I CMAS001 Topology Connect for AOR001 Complete - APPLID(AOR001) CICSplex(PRODPLEX).
EYUPM0003I CMAS001 RTA Specification (MYRTASPC) successfully installed for Context(PRODPLEX) Scope(AOR001).

So what happens when transaction ‘ABCD’ is found not to be running in region AOR001? To test that, I canceled all ABCD transactions in the region. Within a few seconds (determined by the Sampling interval and Analysis interval from my definitions) I see the following message in the EYULOG for the CMAS:

03/23/2016 17:12:06 EYUPN0007W CMAS001 Notify created for RTADEF TRANGONE by MRM, Context=PRODPLEX, Target=AOR001, Sev=LW,
03/23/2016 17:12:06 EYUPN0007W CMAS001 Resource=TASK, Key=*, Text=ABCD not running.

In the MVS SYSLOG and CMAS job log, we see similar messages:

+EYUPN0007W CMAS001 Notify created for RTADEF TRANGONE by MRM, 061
Context=PRODPLEX, Target=AOR001, Sev=LW, Resource=TASK, Key=*,
Text=ABCD not running.

Your MVS automations or monitoring package can pickup on this message and alert your operations staff appropriately.

An RTA Alert is also raised that can be viewed in the WUI itself. From the main WUI entry panel, click

Real Time Analysis (RTA)

and then

Outstanding events

The RTA outstanding events view is displayed:

RTA Alerts List

If I click on the event name, I can get more details, such as when the event happened:

RTA Alerts Details

Having been alerted to the event, my observant and efficient operations staff got transaction ‘ABCD’ running once again. Shortly after that occurred (again determined by the Sampling interval and Analysis interval from my definitions) the alert was cleared. The EYULOG shows:

03/23/2016 17:19:56 EYUPN0013W CMAS001 Notify resolved for RTADEF TRANGONE by MRM, Context=PRODPLEX, Target=AOR001, Sev=LW,
03/23/2016 17:19:56 EYUPN0013W CMAS001 Resource=TASK, Key=*, Text=ABCD running again.

A similar message is in the job log and SYSLOG. The alert also is cleared from the RTA outstanding events view in the WUI.

The power of the RTA component of CICSPlex SM is waiting for you to explore! I welcome any questions or comments on this, as well as suggestions for future topics in CICSPlex SM that you would like to see.

13 comments on"Using CICSPlex SM RTA to monitor critical tasks in CICS"

  1. Dieter Lefere April 14, 2016

    Hello Greg,

    very interesting article. I have one question : can this be configured useing the CICSExplorer ?

    Kind regards

    Dieter Lefere

    • Hi Dieter. Thanks for the comment. Unfortunately the CICS Explorer does not support creating these definitions at this time. Development is updating the Explorer all the time with new features though. The best thing for you to do would be to open an RFE for the Explorer and list the things you want to have ported over from the WUI. Having customer input is so important when we go to prioritize what gets put in next. You can open the RFE at https://www.ibm.com/developerworks/rfe/

  2. Theresa Hamilton May 03, 2016

    Thanks Greg. I can see the need to make sure that a certain transaction id is always running CICS but can you tell me what other things can be monitored using CICSPlex SM?

    • Good question Theresa. CPSM EVALDEFs can be setup to monitor a wide variety of resources. The EVALDEF field ‘Resource table’ in the above example was for the TASK table. If you click on the pencil selector icon next to this field, the WUI will present you with a list of various tables you can select from (141 of them as of CPSM 5.2) Some of them are CICSDSA, CICSRGN, DB2CONN IPCONN, LOCFILE, LOCTRAN, MVSESTG, PROGRAM, TERMNL or WEBSERV. A full list of these tables, the fields in them and what they represent can be found in the Knowledge Center here

      http://www.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.doc/eyua6/cdmdet_all.html?lang=en

  3. Emilio Guarnieri January 23, 2017

    Hello Greg, very useful example. It would be nice to have a list of the possible actions: is it, for example, possible to quiesce a WLM target CICS region when the DB2CONN changes fron CONNECTED to NOTCONNECTED? I am not asking for a consultancy, it is just an example, but it is not really clear what can be performed by CPSM directly and what must be delegated to the automation tools…

    • Hello Emilio. CPSM RTA can take action as well as issue messages, but the actions it takes are directed to the table being monitored in the EVALDEF. The action to be performed is specified in the “Modification String” field. For example, you could monitor the TASK table for an active task, and if it used more than a preset amount of CPU, you could have CPSM issue a SET TASK PURGE. What you could not do is perform an action on a different CPSM table than the one you were monitoring, such as quiesce it as a target region. To do that you would either need to write your own CPSM API program or use an automation tool. The specific action that can be taken on each table is listed at the beginning of each table described in the “CICSPlex SM resource tables” link above. For example, the TASK table described here http://www.ibm.com/support/knowledgecenter/en/SSGMCP_5.2.0/com.ibm.cics.ts.doc/eyua6/TASKtab.html shows that it supports PURGE, FORCEPURGE and KILL actions.

  4. Edwin Decker March 28, 2017

    If I make a change to the CICS system definition that is already connected to an RTASPEC , the Severity for CICS-stalled event field. Will the change get picked up autmatically or do I have to reinstall the RTASPEC or something else?
    thanks,
    Ed

    • Hi Ed. Good question. However, you are asking about how a CSYSDEF change would affect the SAM (System Availability Monitoring) part of CPSM. The stall detection (and thus severity of a stall) is part of SAM. And while SAM does piggyback on some of the RTA architecture, it’s not related to this article.
      In general, a definitional change does not affect things in the active system – you would have to cycle the region to have it take effect, or use the MAS view to also change the active environment. But there are some exceptions. You should open a question with the CICS Q&A group with specifics and it can be answered through that channel. Regards, Greg

  5. John Carini July 27, 2018

    Hi Greg,
    Could this script be used to issue an eval/action where a transaction id be disabled in CICS, if the tran exceeds a 1000 transaction count @ specified time of day?
    regards,
    John Carini
    UPS

  6. Varma Nadimpally January 23, 2019

    Great article. Thanks a lot. Do you have something similar for Work Load Manager Administration View. Please let me know if you have one i am having trouble every time i try to set up something using this view.

    • Thanks for the feedback Varma. We can certainly produce more articles like this, but CPSM Workload Manager is a very broad topic. We would need more details on what specifically you are attempting to do to narrow the focus a bit. If you want to open a PMR with CPSM L2 support to give more details, we could use that as a launch point for authoring a future article.
      Regards,
      Greg Todaro
      IBM CICS/CPSM L2

Join The Discussion

Your email address will not be published. Required fields are marked *