DFHRM0131 for IRC resources then CICS shutdown hangs

This problem sometimes occurs when an incorrect transaction definition for CSKP is being used. CICS creates a transaction definition for CSKP automatically during startup that points to program DFHRMXN3. You shouldn't have a TRANSACTION definition supplied in one of your startup groups.

You can enter "CEMT INQ TRANSACTION(CSKP)" to display the program CSKP is pointing to. If it shows a program other than DFHRMXN3 then it has picked up an incorrect transaction definition for CSKP. It should look something like this:

  Tra(CSKP) Pri( 001 ) Pro(DFHRMXN3  ) Tcl( DFHTCL00 ) Ena Sta
   Prf(DFHCICST) Cda Bel Iso               Bac Wai

Sometimes we see CSKP being associated with an old program called DFHAKP that is no longer used.

You can also enter "CEDA DI TRANSACTION(CSKP) GROUP(*)" to find the group that contains the transaction definition for CSKP.

Not only can an incorrect transaction definition cause CICS shutdown to hang, it also means that CICS is not taking activity keypoints which can lead to your MVS log streams filling up.

If you wanted to see the problem in a dump of CICS, you can format the transaction related control blocks with "VERBX DFHPDxxx 'PCT' " (where xxx is your release level of CICS like 690 for CICS TS 5.2).

Then do a "FIND" command for CSKP. This should get you to the TXDINST control block for CSKP.

Offset x'60' shows the initial program associated with the CSKP transaction which you can easily identify in the eyecatcher. If you see something like DFHAKP below, this is the problem (it should be DFHRMXN3):

 TXDINST.CSKP 179159C0 TXD current instance
 
   Definetime  UNKNOWN                Definesource GROUPNM
   Changetime  UNKNOWN                Changeusrid  UNKNOWN     Changeagent  UNKNOWN
   Installtime 2015/11/01 01:58:22    Installusrid CICSOP      Installagent GRPLIST
 0000  01106EC4 C6C8E7D4 E3E7C4C9 D5E2E340  C3E2D2D7 17914660 00000000 179159C0  *..>DFHXMTXDINST CSKP.j.-.....j.{*
 0020  00000443 00000001 80020000 00000000  00000000 1790E780 00000443 00000000  *......................X.........*
 0040  00000000 00000001 17911CB0 00000000  00000000 00000000 00000000 00000000  *.........j......................*
 0060  C4C6C8C1 D2D74040 C4C6C8C3 C9C3E2E3  00000000 0102FF01 00000000 00000000  *DFHAKP  DFHCICST................*
 0080  01010101 00000000 00000000 02020202  00000000 00000000 00000000 00000000  *................................*
 00A0  00000000 00000000 02020202 00000000  00000000 02020201 01020100 40404040  *............................    *
 00C0  40404040 02000000 00000000 C7D9D6E4  D7D5D440 00000000 00000000 00000000  *    ........GROUPNM ............*

The group that the incorrect CSKP definition is coming from is identified by the Definesource statement. From this example, the group would be GROUPNM.