Question & Answer
Question
I have had my test z/OS Connect V2 Enterprise Edition (zCEE) servers up for several weeks, but for some unknown reason now when I startup any of my zCEE servers (could be one configured as a CICS-WOLA Service provider, WMQ service provider, etc.), they all return the following FFDC message in the messages.log after startup.
[4/25/17 10:39:58:389 EDT] 00000046 com.ibm.ws.logging.internal.impl.IncidentImpl I FFDC1015I: An FFDC Incident has been created: "java.util.ServiceConfigurationError: javax.servlet.ServletContainerInitializer: Provider org.glassfish.jersey.servlet.init.JerseyServletContainerInitializer not found com.ibm.ws.webcontainer.osgi.DynamicVirtualHost startWebApp" at ffdc_17.04.25_10.39.58.0.log
The real problem is that as soon as I submit an http request to list the zosConnect/services URI, it fails with these messages and exception stack.
[4/25/17 10:40:26:443 EDT] 00000046 com.ibm.ws.webcontainer.webapp E SRVE0266E: Error occured while initializing servlets: java.lang.ClassCastException: com.ibm.ws.webcontainer.osgi.servlet.ServletWrapper incompatible with com.ibm.ws.webcontainer.extension.WebExtensionProcessor at com.ibm.ws.webcontainer.webapp.WebApp.createServletWrappers(WebApp.java:1363) at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinish(WebApp.java:1115) at com.ibm.ws.webcontainer.webapp.WebApp.initialize(WebApp.java:1035) at com.ibm.ws.webcontainer.webapp.WebApp.initialize(WebApp.java:6545) at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost.startWebApp(DynamicVirtualHost.java:466) at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost.createRunnableHandler(DynamicVirtualHost.java:264) at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost.createRunnableHandler(DynamicVirtualHost.java:329) at com.ibm.ws.http.internal.VirtualHostImpl.discriminate(VirtualHostImpl.java:251) at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:301) at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:471) at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:405) at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:285) at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:256) at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1043) at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:709) at com.ibm.ws.channel.ssl.internal.SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:397) at com.ibm.ws.channel.ssl.internal.SSLUtils.handleHandshake(SSLUtils.java:976) at com.ibm.ws.channel.ssl.internal.SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:88) at com.ibm.ws.tcpchannel.internal.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:157) at com.ibm.io.async.AbstractAsyncFuture$WorkCallback.run(AbstractAsyncFuture.java:385) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:785)
I have been scanning the jar files in my zCEE directory paths, looked in my server.xml, etc, and I see no references to the org.glassfish.jersey.servlet.init.JerseyServletContainerInitializer class.
My messages.log shows this environment information.
product = WebSphere Application Server 16.0.0.3, WAS FOR Z/OS 16.0.0.3, z/OS Connect 02.00.07 (wlp-1.0.14.cl160320160831-1555) wlp.install.dir = /zosconnect/v2r0/seta/wlp/ server.config.dir = /var/zosconnect/servers/zconCSQ2/ java.home = /java/java864UI46007/J8.0_64 java.version = 1.8.0 java.runtime = Java(TM) SE Runtime Environment (pmz6480sr4fp2-20170322_01 (SR4 FP2)) os = z/OS (02.02.00; s390x) (en_US)
I will attach the .zip dump of my zCEE server.
Any suggestions as to what is causing this would be appreciated.
Answer
The diagnostic files in the Introspection folder in the dump had no references to the org.glassfish.jersey. There were some for org.glassfish.json, but nothing for jersey. The JavaRuntimeEnvironment.txt file however showed many additional jar files had been appended to the class path for the zCEE server to support Spark. The additional jars appear to have been added by an updated zosconnect start script in the -cp command.
Remove the Spark jar files on the classpath. The zosconnect start script only needs the following on the classpath.
-Djava.class.path=/zosconnect/curr/bin/tools/zosconnect.jar:/usr/include/java_classes/ifaedjreg.jar
In this case, the customer confirmed that after removing the appended jar files from the -cp command within the updated /usr/lpp/IBM/zosconnect/v2r0/bin/zosconnect script, the zCEE server started up without the FFDC message and the request to list services no longer returned any errors and returned the list of services.
If your site has no additional jar files on the classpath, see MustGather Classloader problems for WebSphere Application Server Liberty Profile which describes how to obtain the class loader trace for Liberty
In the zCEE server JCL //STDENV append -verbose:class as follows:
JVM_OPTIONS=-Duser.timezone=US/Eastern -verbose:class
Update your server.xml to add this tracing.
Product Synonym
CICS/TS CICSTS CICS TS CICS Transaction Server;zCEE
Was this topic helpful?
Document Information
Modified date:
13 February 2024
UID
dwa1374568