High CPU usage and high EXCP count in "idle" z/OS Connect EE server

There are several monitors within a z/OS Connect EE server that poll resources periodically looking for updates or new resources. The CICS Service Provider using IPIC connections, also periodically heartbeats the CICS region connections. Both of those can drive up CPU and EXCP counts.

If the high CPU and EXCPs only occur when the CICS Service Provider is initializing, that could be due to security SSL handshake activity on all the configured connections. You can check your server.xml (as well as any include xml files) for polling activity and pollingRate for the following configuration elements. If any of the updateTrigger attributes defaults or specified "polled" is causing too much CPU or EXCPs, consult the IBM documentation Configuration elements for those elements to determine other values that would reduce the polling rate or disable the polling altogether and consider alternatively manually refreshing the configuration on-demand using the command /MODIFY <jobname>.<identifier>,ZCON,REFRESH.

Here are the default update methods and for z/OS Connect EE configuration elements:

• config element defaults to: monitorInterval="500ms" updateTrigger="polled"

• applicationMonitor element defaults to: pollingRate="500ms" updateTrigger="polled"

• keyStore element defaults to: updateTrigger="mbean" (applicable for some security configurations)

• zosconnect_zosConnectAPIs element defaults to: pollingRate="5s" updateTrigger="disabled"

• zosconnect_services element defaults to: pollingRate="5s" updateTrigger="disabled"

• zosconnect_apiRequesters element defaults to: pollingRate="5s" updateTrigger="disabled"

• zosconnect_policy element defaults to: pollingRate="1m" updateTrigger="disabled"

• zosconnect_zosConnectDataXform element defaults to: pollingRate="2s" updateTrigger="polled"

By default, the only ones that are not updateTrigger="disabled" are for 'config', 'applicationMonitor' (which should not be used), and 'zosconnect_zosConnectDataXform'.

If you update your server.xml to disable all updateTrigger attributes in all the configuration elements you use, except for the config element where updateTrigger="mbean", then with the zosConnectCommands feature...

  <feature>zosconnect:zosConnectCommands-1.0</feature>

...you should be able use the command...

 MODIFY <jobname>.<identifier>,zcon,refresh

This should allow you to dynamically update the server artifacts.

Then update the attribute values of these configuration elements as follows:

• config element set: updateTrigger="mbean"

• applicationMonitor element set: updateTrigger="disabled

• zosconnect_zosConnectAPIs element set: updateTrigger="disabled" (the default)

• zosconnect_services element set: updateTrigger="disabled" (the default)

• zosconnect_apiRequesters element set: updateTrigger="disabled" (the default)

• zosconnect_policy element set: updateTrigger="disabled" (the default)

• zosconnect_zosConnectDataXform element set: updateTrigger="disabled"

See The MODIFY command and REFRESH MODIFY command syntax in the z/OS Connect EE V3 IBM documentation for a description and the syntax of the MODIFY command.

If using the CICS service provider, it has a default heartbeatinterval of 30s (second precision). However, you should review the heartbeatinterval setting to ensure it is not too low or that the time unit qualifier has not been omitted. For example, if you specify "heartbeatinterval=15" (no second or millisecond time unit qualifier) the default time unit is in milliseconds and will result in high CPU and EXCP counts in an idle server if CICS IPIC connections have been established.

This heartbeatinterval is described in topic Configuration elements of the z/OS Connect EE documentation.

If updating the heartbeatinterval does not change the observed high-CPU, you might also want to try modifying the heartbeatInterval attribute to a value of 0 to disable the heartbeats and monitor the CPU using SDSF DISPLAY ACTIVE.

If after setting heartbeatInterval=0 in the zosconnect_cicsIpicConnection config element in the server.xml, you still see high CPU in the idle z/OS Connect EE server and all file monitors have been disabled, update your server.xml to add tracing for:

  <logging traceSpecification="zosConnectFileMonitor=all"/>

After you restart the z/OS Connect EE server with the --clean option, when you observe high CPU and high EXCP activity, enter command:

 /MODIFY <server>,DUMP

to produce a compressed .zip dump and contact IBM Support for assistance.

Bill Bulfin
IBM zCEE L2 Support