Confluence going down: ERROR [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [engine.jdbc
Asked by ISS Service:

Greetings, from a few weeks ago, our Confluence environment crashes once or twice a day, and we haven't been able to sort out why.  

This is an extract of the log in wich one of the issues took place:

```
2020-02-07 10:02:33,676 WARN [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool@6f5fc0cb -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:36)
at bucket.core.persistence.hibernate.ConfluenceSharedConnectionProvider.getConnection(ConfluenceSharedConnectionProvider.java:34)
at com.atlassian.hibernate.adapter.adapters.ConnectionProviderV5Adapter.getConnection(ConnectionProviderV5Adapter.java:32)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:505)
at com.atlassian.hibernate.adapter.adapters.session.SessionV2Adapter.connection(SessionV2Adapter.java:160)
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy34.getTransaction(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy85.getPropertySet(Unknown Source)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getUserPreferences(DefaultDarkFeaturesManager.java:55)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getUserEnabledFeatures(DefaultDarkFeaturesManager.java:47)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getDarkFeatures(DefaultDarkFeaturesManager.java:99)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getSiteDarkFeatures(DefaultDarkFeaturesManager.java:86)
at sun.reflect.GeneratedMethodAccessor454.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy40.getSiteDarkFeatures(Unknown Source)
at com.atlassian.confluence.mail.notification.listeners.NotificationApiDarkFeature.isEnabled(NotificationApiDarkFeature.java:62)
at com.atlassian.confluence.notifications.impl.NotificationsEventDispatcher.handleEventInternal(NotificationsEventDispatcher.java:103)
at com.atlassian.confluence.notifications.impl.NotificationsEventDispatcher.handleEvent(NotificationsEventDispatcher.java:91)
at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
at com.atlassian.confluence.event.ConfluenceListenerHandlersConfiguration$TimingListenerHandler$1$1.invoke(ConfluenceListenerHandlersConfiguration.java:69)
at com.atlassian.confluence.event.ConfluenceEventDispatcher$VCacheRequestContextRunnableFactory$1.lambda$run$0(ConfluenceEventDispatcher.java:93)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:50)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:49)
at com.atlassian.confluence.event.ConfluenceEventDispatcher$VCacheRequestContextRunnableFactory$1.run(ConfluenceEventDispatcher.java:93)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:105)
at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:64)
at com.atlassian.confluence.user.listeners.UserSessionExpiryListener.sessionDestroyed(UserSessionExpiryListener.java:46)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:837)
at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:695)
at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:666)
at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:651)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5680)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
at java.lang.Thread.run(Thread.java:745)
2020-02-07 10:02:33,686 ERROR [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [engine.jdbc.spi.SqlExceptionHelper] logExceptions An SQLException was provoked by the following failure: java.lang.InterruptedException
2020-02-07 10:02:33,693 WARN [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@38373e8f took 96095 ms (warning threshold is 5000 ms)
```

Also, we have hundres of Broken pipe warnings (maybe they are producing memory leaks?)

and some errors of GC overhead limit exceeded around the same time of the failures.

```
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:140)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:198)
at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:42)
at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:321)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:93)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
at org.apache.coyote.Response.doWrite(Response.java:491)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
... 340 more
2020-02-07 09:59:53,977 WARN [http-nio-8090-exec-33] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1581065888227, threadId=2372, threadName='http-nio-8090-exec-33', userId='xxxxx', type='web-request', summary='/rest/inlinecomments/1.0/comments?containerId=423302281&_=1581065867760'}
-- referer: https://confluence.xxxxxx.com/pages/viewpage.action?pageId=423302281 | url: /rest/inlinecomments/1.0/comments | traceId: 601d20da5498bb64 | userName: xxxxx
2020-02-07 10:00:16,643 WARN [http-nio-8090-exec-56] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@7006d554 took 38149 ms (warning threshold is 5000 ms)
-- url: /login.action | traceId: ac2874dffbb064e0 | userName: anonymous
2020-02-07 10:00:56,903 WARN [http-nio-8090-exec-49] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1581065888197, threadId=2391, threadName='http-nio-8090-exec-49', userId='xxxxx', type='web-request', summary='/plugins/pagetree/naturalchildren.action?decorator=none&excerpt=false&sort=position&reverse=false&disableLinks=false&expandCurrent=true&hasRoot=true&pageId=520946601&treeId=0&startDepth=0&mobile=false&treePageId=549555039&_=1581065880756'}
-- url: /plugins/pagetree/naturalchildren.action | traceId: 7ab747d71365fd89 | userName: xxxxx | referer: https://confluence.xxxxx.com/display/PAR/DC.+Plantilla+plan+de+Medios
2020-02-07 10:00:56,909 ERROR \[http-nio-8090-exec-12\] \[common.error.jersey.ThrowableExceptionMapper\] toResponse Uncaught exception thrown by REST service: GC overhead limit exceeded
-- referer: https://confluence.xxxxx.com/display/CIR/Grandes+Viajes%3A+Circuitos%2C+Combinados+y+Rutas+en+Coche | url: /rest/com.atlassian.confluence.contributors/1.0/contributors | traceId: 8cdc338f5fde9aa0 | userName: xxxxx
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-02-07 10:00:56,913 INFO [http-nio-8090-exec-31] [crowd.directory.ldap.SpringLdapTemplateWrapper] call Timed call for search with handler on OU=World,DC=CORP,DC=xxxxx,DC=COM took 55091ms
2020-02-07 10:00:56,918 ERROR \[http-nio-8090-exec-13\] \[common.error.jersey.ThrowableExceptionMapper\] toResponse Uncaught exception thrown by REST service: GC overhead limit exceeded
-- referer: https://confluence.xxxxx.com/display/CIR/Grandes+Viajes%3A+Circuitos%2C+Combinados+y+Rutas+en+Coche | url: /rest/com.atlassian.confluence.contributors/1.0/contributors | traceId: 6624b401ad6ac5d0 | userName: xxxxx
java.lang.OutOfMemoryError: GC overhead limit exceeded
```

<br />

I searched a bit, but i'm still not versed enough with linux process management to pinpoint the cause.  

Any help would be appreciated.  

Thank you.


Answers:
author: ISS Service
body: Anyone?

comments: author: papireddy
body: we are also facing the same issue. Any suggestion will be appreciated.


Tags: confluence-server

linux

server