Failed to connect other servers with errors "Reason: Too many open files Remote Host: XXXX Remote Port: NNNN"
Sometimes when connecting to other servers, failed with below errors for example:
[2/20/19 17:31:54:059 GMT+07:00] 0000160f SelectionMana W WWLM0061W: An error was encountered sending a request to cluster member {CELLNAME=SVNCFAXPIDAWAS1Cell01, CLUSTERNAME=LSDCluster, lsdMemberDistinction=SVNCFAXPIDAWAS1} and that member has been marked unusable for future requests to the cluster "", because of exception: org.omg.CORBA.COMM_FAILURE: CONNECT_FAILURE_ON_SSL_CLIENT_SOCKET - JSSL0130E: java.io.IOException: Signals that an I/O exception of some sort has occurred. Reason: Too many open files Remote Host: SVNCFAXPIDAWAS1 Remote Port: 9205 vmcid: 0x49421000 minor code: 80 completed: No
How could this happen and how to resolve this?
This means the remote end server responded back that no available file handlers on the remote end host/machine. So, 1. firstly, please figure out which remote host/machine returned this error, 2. and then check the RLIMIT_NOFILE settings on that remote host/machine. If the RLIMIT_NOFILE is small, increased both the soft and hard limit value on OS to larger as 65535 and see if issue resolved: reference: “Guidelines for setting ulimits (WebSphere Application Server)” https://www-01.ibm.com/support/docview.wss?uid=swg21469413 “Insufficient ulimit Value Causes Native OutOfMemory” http://www-01.ibm.com/support/docview.wss?uid=swg21648497 please set both the soft limit and hard limit for RLIMIT_NOFILE to 65535 if the current setting is too small. 3. if the RLIMIT_NOFILE has been large enough, but issue still happens, then please collect lsof data to see if any file handler leak on this remote host/mahine. reference: “Too Many Open Files error message” http://www-01.ibm.com/support/docview.wss?uid=swg21067352 **collect javacore files using command kill -3 PID Also collect lsof output using command for example on linux os: lsof -p [PID] -r [interval in seconds, 1800 for 30 minutes] > lsof.out (To determine if the number of open files is growing over a period of time, issue lsof to report the open files against a PID on a periodic basis.It is best to capture lsof several times to see the rate of growth in the file descriptors. ) Also collect procfiles output using command procfiles -n [PID] > procfiles.out Finally upload all the javacore files, server logs, lsof and profiles output for the servers.
The key point is to check the settings and leak on remote end machine/host, although in some cases maybe they are on same machine. Many cases they are not on same machine.