Why do I keep getting CTG9630E messages when I connect to CICS Transaction Gateway?

CTG9630E is a generic 'connection failed' message. It can come from a variety of causes, both internal, such as configuration issues, and external, as when there are network issues. The following checklist can be used to catch the most common causes.

2. Is the gateway listening on the port you expect? Sometimes the gateway is running but the port it listens on does not match what the application or application server expects. Check the gateway log or STDERR to see if any port conflicts were reported during startup with either the TCP or SSL listener.

A 'netstat -a' command at a command prompt will show you all the ports in use on the system and the processes that are using them. If CICS TG is not listed it suggests the gateway is not active or not able to receive connections. If it is listed with another port you'll need to adjust your application to match.

In an application server, like WebSphere, check the ConnectionURL and Port settings in the connection factory definition to make sure they match the gateway's ctg.ini file settings for protocol@tcp.parameters or protocol@ssl.parameters. If no port is specified for TCP/IP connections, CICS TG will use 2006 as a default.

3. Are there any network issues? If the network had problems it may have cut the gateway's connection without warning. When that happens the gateway is often unaware that any work is trying to reach it and won't log any errors. Sometimes recycling the gateway is all you need to do to re-establish network connections even if the problem occurred outside of the gateway's control.

We often see issues in this area where hardware (switches or routers) or software (firewalls) time out the connection without warning. If the problem most often occurs after long periods of non-use, this is the first place to check. Occasionally, we see connections dropped by the network after a set amount of time regardless of how much traffic the gateway is receiving.

4. What version of CICS resource adapters are connecting to the gateway? If the application or application server’s version of the adapters -- usually cicseci.rar or cicseciXA.rar -- is higher than the version of the gateway then the gateway will reject the connection. So, if you’re running, say, CICS TG v9.0, you want to avoid using any adapters at v9.1 or up on the application side.

If deploying our resource adapter in WebSphere, care must be choose the right adapter for the versions of WebSphere and CICS TG. We have a TechNote which explains the best adapter to deploy.

5. What connection model is your application using? In a managed model, where the application server manages a pool of connections to the gateway, connections are reused as soon as they become available. So long as the application or application server(s) are configured not to exceed the gateway’s MaxConnect setting there should be few issues.

Problems can occur when an unmanaged model is used. When stand-alone applications are used or application servers are used but don’t pool connections, the application will be responsible for opening AND closing each connection. Very often it is assumed that the gateway will close the connection automatically. While the gateway will close idle connections, the default time for this is 600000ms (10 minutes), based on the idletimeout setting in ctg.ini. That can be a very long time for a connection to remain idle, particularly during times of heavy load.

The gateway will keep this connection open in case the original application wants to reuse it, so will not make this available for any requests that arrive from different applications. This may not pose a problem during testing where there may not be as much activity, but in production, during periods where the load is heavy, idle connections can build up to the point where the gateway’s limit of incoming connections is exhausted. The gateway's Connection Manager is limited in the total number of connections it can receive by the MaxConnect setting. Once that limit has been reached it will be forced to reject connections until either some applications close their connections or idle ones begin to time out.

While MaxConnect can be increased to allow for more connections, care must be taken not to set this too high or the gateway will exhaust the amount of memory its JVM can provide. When that happens, the JVM and gateway crash with an OutOfMemory error.

To avoid running out of connections it is best to make sure all applications issue a close() on the connection once they are done using it. Additionally, idletimeout can be reduced to free connections faster. It should be noted that this setting will not affect connections where the gateway is expecting work to return from CICS unless ‘dropworking’ is also specified in the ini file. Without that option enabled the gateway will only close connections where work has returned and no further action is seen from the application side.

.

Gathering More Data On Connection Failures

If the above suggestions don't help it is useful to get trace data from both sides of the connection during a failure. With this level of detail we can see if connections are reaching the gateway and if there are any specific errors that are not typically logged. We have a TechNote which explains how to obtain CICS TG traces and dumps.

On the client side, the type of trace needed depends on the type of application. Java Client Application Trace or .NET Client Trace are the most common. If the application runs under an application server the Java Client Application Trace settings will need to be enabled in the JVM's generic options before the JVM is started.

On the gateway side a debug trace will show connections received from the client side.

The traces must be taken concurrently so both sides of the connection can be observed.