Introduction

First and foremost, it is important not to ignore error messages and to pay attention to them when you see them. There are many different types of DSI messages especially if you include WebSphere eXtreme Scale (WXS). WXS is used to distribute the runtime across a cluster of servers. DSI and WXS use exceptions to draw attention to a disruption of its servers that run in a cluster. Exceptions often indicate that a system is not in a stable condition, but they can also occur during a transitional period from which the system can recover. When a system does recover, some exceptions can be identified as false positive errors (wrongly indicating that a problem is present). In some cases, the exceptions can be used to help you to avoid an unrecoverable situation in the future.

The severity of the exceptions can be grouped into the following categories. The severity of the exceptions increase the lower down the list you go.

  1. Indicates a given condition exists, when it does not
  2. Worth following up on to know why it occurred and how to prevent it
  3. Temporary issue that might repeat and cause problems later on
  4. Likely configuration and/or start up error

1. Indicates a given condition exists, when it does not

1.1. Server versions are incompatible

com.ibm.websphere.objectgrid.ObjectGridRuntimeException "Stack Dump = com.ibm.websphere.objectgrid.ObjectGridRuntimeException: DefaultDomain:com.ibm.ia:iaMaps:2 com.ibm.websphere.objectgrid.TargetNotAvailableException: Retrying the request expired, the current timeout is set to 30000 (ms). DefaultDomain:com.ibm.ia:iaMaps:2 forward: true num fowards:70 lastIP: xxxx.xxxx.com last request started at 05/02/18 08:23:20.200 GST expire time set at 05/02/18 08:23:49.691 GST Number retries 2 Number retries 2
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessageInternal(XIOClientCoreMessageHandler.java:963)
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:144)
at com.ibm.ws.objectgrid.client.CommonClientCoreMessageHandler.sendGetRequest(CommonClientCoreMessageHandler.java:229)
at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1953)
at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:497)
at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844)
at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380)
at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978)
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451)
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411)
at com.ibm.ia.cluster.ClusterBean.getStoredCompatibilityMode(ClusterBean.java:1520)
at com.ibm.ia.cluster.ClusterBean.doCompatibilityCheck(ClusterBean.java:1463)
at com.ibm.ia.cluster.ClusterBean.checkCompatibility(ClusterBean.java:1422)
at com.ibm.ia.cluster.ClusterBean.doTokenUpdate(ClusterBean.java:900)
at com.ibm.ia.cluster.ClusterBean.transactionEnd(ClusterBean.java:814)
at com.ibm.ws.objectgrid.OGEventListenerHandler.fireTransactionEnd(OGEventListenerHandler.java:539)
at com.ibm.ws.objectgrid.SessionImpl.afterCompletion(SessionImpl.java:2555)
at com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:2298)
at com.ibm.ia.cluster.ClusterBean$1.call(ClusterBean.java:641)
at com.ibm.ia.cluster.ClusterBean$1.call(ClusterBean.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:267)
at com.ibm.ia.cluster.ClusterBean.setServerToken(ClusterBean.java:657)
at com.ibm.ia.cluster.ClusterBean.activate(ClusterBean.java:225)
at Proxy993908b1_b421_462b_9e5c_4d9aac3ceee5.activate(Unknown Source)
at Proxycc18703d_0d76_4185_acee_a9d5d92083f0.activate(Unknown Source)
at com.ibm.ia.wxs.WXSServerManager.shardActivated(WXSServerManager.java:161)
at Proxyfda1ec54_3147_4020_9526_1b33e485cf02.shardActivated(Unknown Source)
at com.ibm.ia.wxs.GridServiceManagerCallback.shardActivated(GridServiceManagerCallback.java:87)
at com.ibm.ws.objectgrid.OGEventListenerHandler.fireActivated(OGEventListenerHandler.java:587)
at com.ibm.ws.objectgrid.ObjectGridImpl.activate(ObjectGridImpl.java:1215)
at com.ibm.ws.objectgrid.ObjectGridImpl.activate(ObjectGridImpl.java:1003)
at com.ibm.ws.objectgrid.replication.PrimaryShardImpl.activateObjectGrid(PrimaryShardImpl.java:6767)
at com.ibm.ws.objectgrid.replication.PrimaryShardActor.activateObjectGrid(PrimaryShardActor.java:441)
at com.ibm.ws.objectgrid.replication.ReplicatedPartition.becomePrimary(ReplicatedPartition.java:1381)
at com.ibm.ws.objectgrid.replication.ReplicatedPartition.processPrimaryWork(ReplicatedPartition.java:641)
at com.ibm.ws.objectgrid.replication.ReplicatedPartition.handleMigrationEvent(ReplicatedPartition.java:542)
at com.ibm.ws.objectgrid.server.impl.ShardImpl.handleRoleTransition(ShardImpl.java:338)
at com.ibm.ws.objectgrid.server.impl.ShardActor.handleRoleTransition(ShardActor.java:1097)
at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.doContainerWork(ObjectGridContainerImpl.java:889)
at com.ibm.ws.objectgrid.server.container.ContainerActor$2.run(ContainerActor.java:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)
Caused by: com.ibm.websphere.objectgrid.TargetNotAvailableException: Retrying the request expired, the current timeout is set to 30000 (ms). DefaultDomain:com.ibm.ia:iaMaps:2 forward: true num fowards:70 lastIP: xxxx.xxxx.com last request started at 05/02/18 08:23:20.200 GST expire time set at 05/02/18 08:23:49.691 GST Number retries 2
at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessageInternal(XIOClientCoreMessageHandler.java:744)
... 42 more

Diagnosis: This runtime exception reports a failed operation to check that all the cluster members have a compatible DSI version. The exception is known to occur when a system is temporarily overused or there is a temporary issue. It occurs after the commit of the transaction, so it does not affect the consistency of the DSI data. If the check fails and throws an exception when the system temporarily prevents its success, the DSI cluster is not impacted.

1.2. Unable to calculate a global aggregate

com.ibm.websphere.objectgrid.ObjectGridException "Stack Dump = com.ibm.websphere.objectgrid.ObjectGridException: java.lang.InterruptedException
at com.ibm.ws.objectgrid.map.GetNextKeyQueue.atomicWait(GetNextKeyQueue.java:348)
at com.ibm.ws.objectgrid.map.GetNextKeyQueue.next(GetNextKeyQueue.java:236)
at com.ibm.ws.objectgrid.map.BaseMap.getNextKey(BaseMap.java:5611)
at com.ibm.ws.objectgrid.DiffMap.getNextKey(DiffMap.java:3116)
at com.ibm.ws.objectgrid.ObjectMapImpl.getNextKey(ObjectMapImpl.java:2099)
at com.ibm.ia.wxs.WXSTransactionStore.take(WXSTransactionStore.java:1300)
at com.ibm.ia.runtime.PartialResultQueueService.take(PartialResultQueueService.java:95)
at com.ibm.ia.runtime.PartialResultQueueService.take(PartialResultQueueService.java:1)
at com.ibm.ia.runtime.BaseGetNextPartialResult.run(BaseGetNextPartialResult.java:100)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
at java.lang.Thread.run(Thread.java:785)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1040)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1329)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:410)
at com.ibm.ws.objectgrid.map.GetNextKeyQueue.atomicWait(GetNextKeyQueue.java:336)
... 13 more

Diagnosis: It is a temporary failure, DSI will retry the operation later. If the global aggregate feature of DSI is not used, it will not have any impact.

1.3. Shard unable to serve requests

com.ibm.ws.xsspi.xio.exception.TransportException "Stack Dump = com.ibm.ws.xsspi.xio.exception.TransportException$Transient [originating=10.50.88.166:7118;exid=0]: Partition 10 for com.ibm.ia in map set iaMaps is temporarily not taking new transactions on xxxx.xxxx.com-ContainerSrvr03_C-13 
at sun.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:436) 
at com.ibm.ws.xsspi.xio.exception.XIOExceptionFactory.createException(XIOExceptionFactory.java:58) 
at com.ibm.ws.xsspi.xio.exception.ServerExceptionFactory.createTransient(ServerExceptionFactory.java:156) 
at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.createInitialReadWriteTransaction(ServerCoreEventProcessor.java:734) 
at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processGetNextRequest(ServerCoreEventProcessor.java:1359) 
at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processClientServerRequest(ServerCoreEventProcessor.java:2679) 
at com.ibm.ws.objectgrid.server.impl.ShardImpl.processMessage(ShardImpl.java:1469) 
at com.ibm.ws.objectgrid.server.impl.ShardActor.handleContainerMessage(ShardActor.java:513) 
at com.ibm.ws.objectgrid.server.impl.ShardActor.receive(ShardActor.java:339) 
at com.ibm.ws.xs.xio.actor.impl.XIOReferableImpl.dispatch(XIOReferableImpl.java:110) 
at com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.java:987) 
at com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable.run(XIORegistryRunnable.java:88) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) 
at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)

Diagnosis: This exception indicates that the client reached the shard, but the shard is in a transitional state, and cannot serve the request. The shard did not complete work to promote to the primary shard. A shard might be experiencing a demotion or the shard might be in the process of stopping.

2. Worth following up on to know why it occurred and how to prevent it

2.1. Event submission rate exceeded capacity

com.ibm.ia.common.EventCapacityExceededException: CWMBD9692E: Event submission rate {0} reached system capacity of {1}
at com.ibm.ia.gateway.impl.SolutionGatewayImpl.submitEvent(SolutionGatewayImpl.java:287)
at com.ibm.ia.gateway.impl.SolutionGatewayImpl.submit(SolutionGatewayImpl.java:204)
at com.ibm.ia.connectivity.inbound.impl.ReusableChannel.submitEvent(ReusableChannel.java:322)
at com.ibm.ia.connectivity.inbound.impl.xml.XMLChannel.submit(XMLChannel.java:185)
at com.ibm.ia.connectivity.inbound.impl.xml.XMLChannel.submit(XMLChannel.java:159)
at com.ibm.ia.connectivity.inbound.impl.GatewayChannelImpl.submit(GatewayChannelImpl.java:94)
at com.ibm.ia.connectivity.inbound.jms.InboundJmsEndpointMDB.onMessage(InboundJmsEndpointMDB.java:197)
at com.ibm.ia.connectivity.inbound.jms.MDBProxycncInboundEndPoint_dfc99253.onMessage(MDBProxycncInboundEndPoint_dfc99253.java)

Diagnosis: This exception reports that a DSI protection mechanism is triggered. The role of the mechanism is to slow the flow of incoming events until the grid is able to process at full capacity again. This might indicate that the number and speed at which events are received is too high for the system in general. Check the configuration of the servers and adjust accordingly.

3. Temporary issue that might cause problems later on

3.1. The grid is not online

com.ibm.websphere.objectgrid.ClientServerLoaderException "Stack Dump = com.ibm.websphere.objectgrid.ClientServerLoaderException: Client Services - Received exception from remote server
 at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.processGetResponse(RemoteTransactionCallbackImpl.java:1384)
 at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.getGetResponseWithAuthenticationRetry(RemoteTransactionCallbackImpl.java:1971)
 at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1957)
 at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:497)
 at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4847)
 at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1388)
 at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978)
 at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451)
 at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411)
 at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458)
 at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412)
 at com.ibm.ia.runtime.stores.TransactionStoreCache.get(TransactionStoreCache.java:103)
 at com.ibm.ia.runtime.EntityStore.getEntity(EntityStore.java:235)
 at com.ibm.ia.runtime.EntityStore.getEntity(EntityStore.java:197)
 at com.ibm.ia.runtime.EntityStore.getEntity(EntityStore.java:190)
 at com.ibm.ia.runtime.CISEntityResolver.resolveEntity(CISEntityResolver.java:116)
 at com.ibm.ia.runtime.CISEntityResolver.getEntity(CISEntityResolver.java:106)
 at com.ibm.ia.runtime.AgentEntityResolver.getEntity(AgentEntityResolver.java:82)
 at com.ibm.ia.runtime.ObjectEntityResolverImpl.resolve(ObjectEntityResolverImpl.java:140)
 at com.ibm.ia.runtime.AgentObjectEntityResolver.resolve(AgentObjectEntityResolver.java:62)
 at com.ibm.ia.model.impl.RelationshipImpl.resolve(RelationshipImpl.java:56)
 at com.ibm.ia.model.impl.Utilities.resolve(Utilities.java:29)
...
 at com.ibm.rules.engine.rete.runtime.state.AbstractConditionExecEnv.evaluateBooleanMethod(AbstractConditionExecEnv.java:75)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorJoinNode.evaluateDiscTest(IlrAbstractGeneratorJoinNode.java:471)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorJoinNode.updateInternal(IlrAbstractGeneratorJoinNode.java:179)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorJoinNode.update(IlrAbstractGeneratorJoinNode.java:211)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractTupleMemNode.notifyUpdate(IlrAbstractTupleMemNode.java:96)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardJoinNode.updateInternal(IlrStandardJoinNode.java:138)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardJoinNode.update(IlrStandardJoinNode.java:162)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractTupleMemNode.notifyUpdate(IlrAbstractTupleMemNode.java:96)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardAlphaNode.update(IlrStandardAlphaNode.java:87)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractObjectMemNode.notifyUpdate(IlrAbstractObjectMemNode.java:74)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorDiscNode.updateInternal(IlrAbstractGeneratorDiscNode.java:256)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorDiscNode.updateEngineData(IlrAbstractGeneratorDiscNode.java:304)
 at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardWorkingMemoryNode.updateEngineData(IlrStandardWorkingMemoryNode.java:119)
 at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.updateData(AbstractReteEngine.java:339)
...
 at com.ibm.rules.engine.rete.runtime.state.AbstractBodyExecEnv.executeBody(AbstractBodyExecEnv.java:82)
 at com.ibm.rules.engine.rete.runtime.state.AbstractNetworkState.executeRule(AbstractNetworkState.java:204)
 at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.executeOneRule(AbstractReteEngine.java:470)
 at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.stepOneRule(AbstractReteEngine.java:632)
 at com.ibm.ia.engine.runtime.impl.EventEngineImpl.executeRules(EventEngineImpl.java:467)
 at com.ibm.ia.engine.runtime.impl.EventEngineImpl.insertEventInternal(EventEngineImpl.java:422)
 at com.ibm.ia.engine.runtime.impl.EventEngineImpl.insertEvent(EventEngineImpl.java:314)
 at com.ibm.ia.agent.BaseRuleAgent.insertEvent(BaseRuleAgent.java:187)
 at com.ibm.ia.agent.RuleAgent.processEvent2(RuleAgent.java:253)
 at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:224)
 at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119)
 at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124)
 at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403)
 at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107)
 at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758)
 at com.ibm.ia.runtime.EventQueueMgr.post(EventQueueMgr.java:220)
 at com.ibm.ia.runtime.EventQueueMgr.post(EventQueueMgr.java:168)
 at com.ibm.ia.runtime.GatewayListener.processEvent(GatewayListener.java:79)
 at com.ibm.ia.runtime.OutboundEventDispatcher.dispatch(OutboundEventDispatcher.java:239)
 at Proxy398b549f_02dc_4448_951c_6f0f0780fbc1.dispatch(Unknown Source)
 at com.ibm.ia.runtime.AgentContext.dispatch(AgentContext.java:229)
 at com.ibm.ia.common.BaseComponent.processOutboundEvents(BaseComponent.java:351)
 at com.ibm.ia.agent.Agent.processOutboundEvents(Agent.java:535)
 at com.ibm.ia.runtime.EventProcessor.onEventDoTimeTriggers(EventProcessor.java:1229)
 at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:754)
 at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:267)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
 at java.lang.Thread.run(Thread.java:785)
Caused by: com.ibm.websphere.objectgrid.TransactionException: com.ibm.websphere.objectgrid.TransactionException:transaction rolled backed by another thread
 at RemovedByServerSerialization.RemovedByServerSerialization(RemovedByServerSerialization:0)
Caused by: com.ibm.websphere.objectgrid.TransactionException: com.ibm.websphere.objectgrid.TransactionException:The current state of the ObjectGrid instance caused the transaction to end.
 ... 1 more
 
 and
 
com.ibm.ia.common.AgentException "Stack Dump = com.ibm.ia.common.AgentException: com.ibm.ia.common.AgentException: com.ibm.ia.common.AgentException: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread 
at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:1093) 
at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467) 
at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186) 
at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119) 
at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124) 
at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403) 
at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107) 
at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758) 
at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.run(FutureTask.java:267) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) 
at java.lang.Thread.run(Thread.java:785) 
Caused by: com.ibm.ia.common.AgentException: com.ibm.ia.common.AgentException: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread 
at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:794) 
at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937) 
... 13 more 
Caused by: com.ibm.ia.common.AgentException: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread 
at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:241) 
at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801) 
at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769) 
... 14 more 
Caused by: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread 
at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:468) 
at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412) 
at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85) 
at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238) 
... 16 more 
Caused by: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread 
at com.ibm.ws.objectgrid.SessionImpl.mapPostInvoke(SessionImpl.java:3724) 
at com.ibm.ws.objectgrid.ObjectMapImpl.postInvoke(ObjectMapImpl.java:883) 
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:476) 
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) 
at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458) 
... 19 more 
Caused by: com.ibm.websphere.objectgrid.TransactionException: The current state of the ObjectGrid instance caused the transaction to end. 
at com.ibm.ws.objectgrid.transaction.TransactionManager.processAsyncRollback(TransactionManager.java:606) 
at com.ibm.ws.objectgrid.transaction.TransactionManager.rollbackOutlierBucket(TransactionManager.java:558) 
at com.ibm.ws.objectgrid.transaction.TransactionManager.rollbackNextBucket(TransactionManager.java:531) 
at com.ibm.ws.objectgrid.transaction.TransactionManager.drainAll(TransactionManager.java:780) 
at com.ibm.ws.objectgrid.transaction.TransactionManager.destroy(TransactionManager.java:399) 
at com.ibm.ws.objectgrid.ObjectGridImpl.destroy(ObjectGridImpl.java:3315) 
at com.ibm.ws.objectgrid.catalog.placement.balance.disk.DiskCatalogUtil.destroyGrid(DiskCatalogUtil.java:176) 
at com.ibm.ws.objectgrid.server.impl.ShardImpl.destroy(ShardImpl.java:800) 
at com.ibm.ws.objectgrid.server.impl.ShardActor.destroy(ShardActor.java:1104) 
at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.deactivate(ObjectGridContainerImpl.java:1343) 
at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.destroyShard(ObjectGridContainerImpl.java:2394) 
at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.destroyShard(ObjectGridContainerImpl.java:2367) 
at com.ibm.ws.objectgrid.server.container.ContainerActor.destroyShard(ContainerActor.java:406) 
at com.ibm.ws.objectgrid.replication.PrimaryShardImpl.destroyShard(PrimaryShardImpl.java:2353) 
at com.ibm.ws.objectgrid.replication.PrimaryShardImpl.abdicateLeadershipRelayReplicas3Common(PrimaryShardImpl.java:2686) 
at com.ibm.ws.objectgrid.replication.PrimaryShardActor.abdicateLeadershipRelayReplicas3Common(PrimaryShardActor.java:308) 
at com.ibm.ws.objectgrid.replication.PrimaryShardActor.abdicateLeadershipReceive(PrimaryShardActor.java:163) 
at com.ibm.ws.objectgrid.replication.PrimaryShardActor.receive(PrimaryShardActor.java:115) 
at com.ibm.ws.xs.xio.actor.impl.XIOReferableImpl.dispatch(XIOReferableImpl.java:110) 
at com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.java:987) 
at com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable.run(XIORegistryRunnable.java:88) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) 
at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)

Diagnosis: The grid is not online. It typically occurs when a shutdown is in progress or in case of major system issue.

4. Likely configuration and/or start up error

4.1. Failed to obtain a database connection

java.lang.reflect.InvocationTargetException "Stack Dump = java.lang.reflect.InvocationTargetException
 at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
 at java.lang.reflect.Method.invoke(Method.java:507)
 at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87)
 at com.ibm.ws.xs.osgi.service.ObjectGridServiceHandler.invoke(ObjectGridServiceHandler.java:76)
 at com.sun.proxy.$Proxy60.populateTXSlots(Unknown Source)
 at com.ibm.ia.persistence.impl.SQLPersistenceManager.get(SQLPersistenceManager.java:644)
 at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
 at java.lang.reflect.Method.invoke(Method.java:507)
 at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87)
 at com.ibm.ws.xs.osgi.service.BackingMapServiceHandler.invoke(BackingMapServiceHandler.java:74)
 at com.sun.proxy.$Proxy427.get(Unknown Source)
 at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844)
 at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380)
 at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978)
 at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451)
 at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411)
 at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458)
 at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412)
 at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85)
 at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238)
 at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801)
 at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769)
 at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937)
 at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467)
 at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186)
 at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119)
 at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124)
 at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403)
 at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107)
 at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758)
 at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:267)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
 at java.lang.Thread.run(Thread.java:785)
Caused by: com.ibm.ia.common.PersistenceException: com.ibm.ia.common.PersistenceSQLException: Connection not available, Timed out waiting for 180001
 at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:160)
 at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:105)
 ... 38 more
Caused by: com.ibm.ia.common.PersistenceSQLException: Connection not available, Timed out waiting for 180001
 ... 40 more

and

com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException "Stack Dump = com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180001
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1442)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3050)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2335)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1121)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:688)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:354)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:325)
at com.ibm.ia.persistence.impl.SQLCallbackImpl.putConnection(SQLCallbackImpl.java:117)
at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:147)
at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:105)
at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87)
at com.ibm.ws.xs.osgi.service.ObjectGridServiceHandler.invoke(ObjectGridServiceHandler.java:76)
at com.sun.proxy.$Proxy60.populateTXSlots(Unknown Source)
at com.ibm.ia.persistence.impl.SQLPersistenceManager.get(SQLPersistenceManager.java:644)
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87)
at com.ibm.ws.xs.osgi.service.BackingMapServiceHandler.invoke(BackingMapServiceHandler.java:74)
at com.sun.proxy.$Proxy427.get(Unknown Source)
at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844)
at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380)
at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978)
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451)
at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411)
at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458)
at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412)
at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85)
at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238)
at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801)
at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769)
at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937)
at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467)
at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186)
at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119)
at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124)
at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403)
at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107)
at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758)
at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
at java.lang.Thread.run(Thread.java:785)

and

java.sql.SQLTransientConnectionException "Stack Dump = java.sql.SQLTransientConnectionException: Connection not available, Timed out waiting for 180001
 at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1100)
 at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:368)
 at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:325)
 at com.ibm.ia.persistence.impl.SQLCallbackImpl.putConnection(SQLCallbackImpl.java:117)
 at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:147)
 at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:105)
 at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
 at java.lang.reflect.Method.invoke(Method.java:507)
 at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87)
 at com.ibm.ws.xs.osgi.service.ObjectGridServiceHandler.invoke(ObjectGridServiceHandler.java:76)
 at com.sun.proxy.$Proxy60.populateTXSlots(Unknown Source)
 at com.ibm.ia.persistence.impl.SQLPersistenceManager.get(SQLPersistenceManager.java:644)
 at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
 at java.lang.reflect.Method.invoke(Method.java:507)
 at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87)
 at com.ibm.ws.xs.osgi.service.BackingMapServiceHandler.invoke(BackingMapServiceHandler.java:74)
 at com.sun.proxy.$Proxy427.get(Unknown Source)
 at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844)
 at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380)
 at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978)
 at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451)
 at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411)
 at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458)
 at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412)
 at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85)
 at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238)
 at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801)
 at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769)
 at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937)
 at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467)
 at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186)
 at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119)
 at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124)
 at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403)
 at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107)
 at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758)
 at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:267)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
 at java.lang.Thread.run(Thread.java:785)
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180001
 at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1097)
 ... 43 more
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180001
 at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1442)
 at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3050)
 at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2335)
 at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1121)
 at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:688)
 at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:354)
 ... 42 more

Diagnosis: These exceptions are indicating that DSI has failed to obtain a database connection.
It might occur for multiple reasons: Insufficient connection pool size, database slowness, or network issues.

4.2. Outbound servers started before runtime servers

com.ibm.ia.connectivity.outbound.sender.PermanentEventSenderException: Unable to access configuration for solution Solution_Name
at com.ibm.ia.connectivity.outbound.sender.impl.SolutionSenders.getConfigurationSource(SolutionSenders.java:359)
at com.ibm.ia.connectivity.outbound.sender.impl.SolutionSenders.getEventSender(SolutionSenders.java:80)
at com.ibm.ia.connectivity.outbound.sender.impl.EventSenderManagerImpl.getEventSender(EventSenderManagerImpl.java:362)
at com.ibm.ia.connectivity.outbound.async.impl.AsynchronousWorkerThread.run(AsynchronousWorkerThread.java:263)
Caused by: com.ibm.ia.connectivity.config.ConfigurationException: Unable to access solution configuration from the grid: Solution_Name
at com.ibm.ia.connectivity.config.grid.impl.ConfiguredGridConnectionManager.getSolutionConfiguration(ConfiguredGridConnectionManager.java:680)
at com.ibm.ia.connectivity.config.grid.impl.ConfiguredGridConnectionManager.getConfigurationSource(ConfiguredGridConnectionManager.java:635)
at com.ibm.ia.connectivity.outbound.sender.impl.SolutionSenders.getConfigurationSource(SolutionSenders.java:355)
... 3 more
Caused by: com.ibm.ia.gateway.GridConnectionException: Grid is not available
at com.ibm.ia.wxs.client.impl.ObjectGridEnvironment.getSolutionStore(ObjectGridEnvironment.java:305)
at com.ibm.ia.gateway.spi.ClientConnectionImpl.getSolutionConfiguration(ClientConnectionImpl.java:169)
at com.ibm.ia.connectivity.config.grid.impl.ConfiguredGridConnectionManager.getSolutionConfiguration(ConfiguredGridConnectionManager.java:668)
... 5 more

Diagnosis: This exception reports an outbound server is started before all of the runtime servers are ready. This is a transient condition until the grid is ready. The error disappears when all of the servers are started. Make sure that you start the servers in the following order: Catalog servers, Runtime servers, Outbound servers, Inbound servers.

Join The Discussion

Your email address will not be published. Required fields are marked *