When itncm.sh script is used with the STOP option, it does not stop ITNCM at all.. Why am I forced to kill ITNCM process or use itncm.sh force to stop ?
When issuing the "itncm.sh stop" command the below error message is displayed and ITNCM is not stopped:
[icosuser@itncm drivers]$ /opt/IBM/tivoli/netcool/ncm/bin/itncm.sh stop
IBM Tivoli Netcool Configuration Manager
----------------------------------------
Stopping GUI Server
Please enter the Intelliden Super User and password if prompted below:
The Netcool Configuration Manager GUI Server has not been stopped
successfully.
Please Re-run using the correct Super User credentials.
[icosuser@itncm drivers]$
You are unable to use the itncm.sh stop or restart options on the ITNCM server. It simply just tries to stop the GUI server but then fails. Using the itncm.sh force option works and then you can start back using itncm.sh start. Issue occurs when running this script using the icosuser ID.
You are unable to use the itncm.sh stop or restart options on the ITNCM server. It simply just tries to stop the GUI server but then fails. Using the itncm.sh force option works and then you can start back using itncm.sh start. Issue occurs only when running this script using the icosuser ID.
Cause : Exception seen related to SSL handshake and when websphere application server is trying to stop.
To resolve this issue first refer to the /eWAS/profiles/RSeries/logs/server1/stopServer.log example /opt/IBM/tivoli/netcool/ncm/eWAS/profiles/RSeries/logs/server1/stopServer.log and look out for errors such as the following snip noted below, when itncm.sh stop is executed:
[6/16/14 14:59:54:558 EEST] 00000000 WSX509TrustMa E CWPKI0311E: The certificate with subject DN CN=itncm, O=IBM, C=US has a start date Sun Jan 13 00:00:04 EET 2030 which is valid after the current date/time. This will can happen if the client's clock is set earlier than the server's clock. Please verify the clocks are in sync between this client and server and retry the request.
[6/16/14 14:59:54:582 EEST] 00000000 WsServerStop E ADMU3002E: Exception attempting to process server server1
[6/16/14 14:59:54:584 EEST] 00000000 WsServerStop E ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host itncm at port 18103.
[6/16/14 14:59:54:585 EEST] 00000000 WsServerStop A ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host itncm at port 18103.
at com.ibm.ws.management.connector.ConnectorHelper.createConnector(ConnectorHelper.java:606)
at com.ibm.ws.management.tools.WsServerStop.runTool(WsServerStop.java:372)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:272)
at com.ibm.ws.management.tools.WsServerStop.main(WsServerStop.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:213)
at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:93)
at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:74)
at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:78)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:68)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:400)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
at org.eclipse.core.launcher.Main.run(Main.java:981)
at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:341)
at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:111)
Caused by: com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host itncm at port 18103.
at com.ibm.websphere.management.AdminClientFactory.createAdminClientPrivileged(AdminClientFactory.java:634)
at com.ibm.websphere.management.AdminClientFactory.access$000(AdminClientFactory.java:125)
at com.ibm.websphere.management.AdminClientFactory$1.run(AdminClientFactory.java:208)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:204)
at com.ibm.ws.management.connector.ConnectorHelper.tryProtocol(ConnectorHelper.java:643)
at com.ibm.ws.management.connector.ConnectorHelper.createConnector(ConnectorHelper.java:575)
... 24 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:44)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:516)
at com.ibm.websphere.management.AdminClientFactory.createAdminClientPrivileged(AdminClientFactory.java:456)
... 30 more
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=racs-itncm, OU=Root Certificate, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=racs-itncm, OU=Root Certificate, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error]
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:409)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.<init>(SOAPConnectorClient.java:222)
... 35 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=itncm, OU=Root Certificate, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=itncm, OU=Root Certificate, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:475)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:451)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:373)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:366)
Steps to resolve :
If you refer to the "stopserver ()" logic routine within the itncm.sh script, you will notice that it stops the presentation server (with embedded websphere appserver) and if the presentation server is not stopped, then the user will not be allowed to enter Intelliden super-user name and password. So basically, the presentation server is not getting stopped gracefully in this instance , hence you are seeing this problem.
Upon reviewing the stopServer.log , you may notice the exception noted below, which is seen to be related to an SSL handshake. When the websphere application server is trying to stop, it is then trying to look for a valid certificate from the context CN=itncm, OU=Root Certificate, O=IBM, C=US, but it is not trusted. This actual SSL handshake is required when a SOAP connection is made to stop the websphere application server:
. .
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=itncm, OU=Root Certificate, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=itncm, OU=Root Certificate, O=IBM, C=US is not trusted; internal cause is:
. .
Please review the following IBM tech-note which relates to websphere for the error code ADMC0016E. The same error code is usually also noticed in the stopServer.log file and relates to the SOAP connection (i.e. SSL handshake happens as part of this SOAP connection): http://www-01.ibm.com/support/docview.wss?uid=swg21667810
This tech-note explains that this might happen when the certificate has expired. So please review this certificate expiry.
However to know the exact reason for SSL handshake failure in this instance you would need to add extra -trace option in the stop Server logic for itncm.sh script as -$WAS_BIN/stopServer.sh server1 -quiet -trace. Once itncm.sh script is changed on the server then run itncm.sh stop again to see more information in the latest stopServer.log file for the exact reason related to certificate error. This will help you debug further if you are unable to resolve the certificate expiry itself. NOTE: Please do revert back to the original itncm.sh script after getting the latest stopServer.log, otherwise the -trace option will generate extra tracing on the server.