Confluence loses connection to Database throughout Day
Asked by Network admin:

Randomly, Confluence has been having connection drops to the PostgreSQL database.

I receive this error:![confluenceerror.PNG](/t5/image/serverpage/image-id/37577i696E036A0BDE665E/image-dimensions/565x296?v=v2 "confluenceerror.PNG")

Whats interesting is that my JIRA set up is running on the same server and same instance of PostgreSQL (9.6) completely fine.

I can provide all the logs you need.

One solution that ive been doing is going to PGadmin(PostgreSQL interface tool) to check on the Databases and seeing if they're connected, which they are. So I leave PGadmin running and i restart Confluence through the service menu in computer management (Windows 10 pro).

Confluence starts to run, however this is a temporary fix, because after a few hours or a day Confluence loses connection to PostgreSQL(Which is still running)

Below is an image of how Confluence DB is configured through PostgreSQL using PGadmin

![ConfluenceDbPG.png](/t5/image/serverpage/image-id/37579i5D359A3FDFA4DE15/image-dimensions/795x381?v=v2 "ConfluenceDbPG.png")

Again, you can see in the image provided above that JIRAdb is there and running perfectly fine.

The two logs below are Catalina Log and Application Log

```
14-Nov-2018 00:12:37.386 SEVERE [http-nio-8090-exec-10] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=404, location=/fourohfour.action]
?java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
?? ?at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
?? ?at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
?? ?at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)
?? ?at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
?? ?at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
?? ?at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
?? ?at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
?? ?at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)
?? ?at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)
?? ?at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)
?? ?at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
?? ?at com.atlassian.confluence.themes.DefaultThemeManager.getThemeSettings(DefaultThemeManager.java:73)
?? ?at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalThemeKey(DefaultThemeManager.java:49)
?? ?at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalTheme(DefaultThemeManager.java:42)
?? ?at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:80)
?? ?at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:53)
?? ?at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)
?? ?at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:32)
?? ?at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)
?? ?at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)
?? ?at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:64)
?? ?at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)
?? ?at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39)
?? ?at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
?? ?at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
?? ?at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
?? ?at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
?? ?at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
?? ?at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
?? ?at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
?? ?at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
?? ?at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)
?? ?at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)
?? ?at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)
?? ?at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
?? ?at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
?? ?at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
?? ?at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
?? ?at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
?? ?at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770)
?? ?at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
?? ?at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
?? ?at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
?? ?at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
?? ?at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
?? ?at java.lang.Thread.run(Thread.java:748)
Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
?? ?at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)
?? ?at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
?? ?at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
?? ?at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)
?? ?at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
?? ?at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
?? ?at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
?? ?at java.lang.reflect.Method.invoke(Method.java:498)
?? ?at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
?? ?at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
?? ?at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)
?? ?at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:344)
?? ?at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313)
?? ?at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:42)
?? ?at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)
?? ?at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
?? ?at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
?? ?at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
?? ?at java.lang.reflect.Method.invoke(Method.java:498)
?? ?at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
?? ?at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
?? ?at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
?? ?at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
?? ?at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
?? ?at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
?? ?at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
?? ?at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
?? ?at com.sun.proxy.$Proxy55.retrieve(Unknown Source)
?? ?at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)
?? ?at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)
?? ?at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)
?? ?at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
?? ?... 70 more
```

```
2018-11-13 23:06:47,676 ERROR [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z4jp4wl1yl4557|6c2fa18e]-HelperThread-#0] [org.postgresql.Driver] connect Connection error: 
org.postgresql.util.PSQLException: FATAL: the database system is starting up
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
?? ?at org.postgresql.Driver.connect(Driver.java:252)
?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
2018-11-13 23:06:47,754 WARN [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z4jp4wl1yl4557|6c2fa18e]-HelperThread-#0] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@72131583 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: the database system is starting up
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
?? ?at org.postgresql.Driver.connect(Driver.java:252)
?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
```


Answers:
author: Network admin
body: > 4-Nov-2018 00:12:37.386 SEVERE \[http-nio-8090-exec-10\] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage\[errorCode=404, location=/fourohfour.action\]  
> java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)  
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)  
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)  
> at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)  
> at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)  
> at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)  
> at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)  
> at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)  
> at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)  
> at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)  
> at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)  
> at com.atlassian.confluence.themes.DefaultThemeManager.getThemeSettings(DefaultThemeManager.java:73)  
> at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalThemeKey(DefaultThemeManager.java:49)  
> at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalTheme(DefaultThemeManager.java:42)  
> at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:80)  
> at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:53)  
> at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
> at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:32)  
> at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
> at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)  
> at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:64)  
> at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)  
> at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39)  
> at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)  
> at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)  
> at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)  
> at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)  
> at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)  
> at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)  
> at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)  
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)  
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)  
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)  
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)  
> at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)  
> at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)  
> at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770)  
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)  
> at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
> at java.lang.Thread.run(Thread.java:748)  
> Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
> at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
> at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)  
> at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
> at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:344)  
> at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313)  
> at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:42)  
> at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)  
> at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)  
> at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)  
> at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
> at com.sun.proxy.$Proxy55.retrieve(Unknown Source)  
> at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)  
> at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)  
> at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)  
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)  
> ... 70 more


author: Nic Brough -Adaptavist-
body: Is there a "disconnect" error in the logs, or is it just a sudden "database not reachable" type thing? Are there significant errors in the logs before such a disconnect?

comments: author: Network admin
body: Under which logs can i find that? I can post the specific logs you're looking for


author: Nic Brough -Adaptavist-
body: I'd expect the errors to go into the application logs, so \<confluence home\>/atlassian-confluence.log is the main one. There might be informative errors in \<confluence install\>/catalina.out too.

Have a look at your system info page to locate those two directories if you don't know them already.


author: Network admin
body: ```
2018-11-13 23:06:47,676 ERROR [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z4jp4wl1yl4557|6c2fa18e]-HelperThread-#0] [org.postgresql.Driver] connect Connection error:
org.postgresql.util.PSQLException: FATAL: the database system is starting up
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
?? ?at org.postgresql.Driver.connect(Driver.java:252)
?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
2018-11-13 23:06:47,754 WARN [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z4jp4wl1yl4557|6c2fa18e]-HelperThread-#0] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@72131583 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATAL: the database system is starting up
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
?? ?at org.postgresql.Driver.connect(Driver.java:252)
?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
```


author: Network admin
body: > 4-Nov-2018 00:12:37.386 SEVERE \[http-nio-8090-exec-10\] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage\[errorCode=404, location=/fourohfour.action\]  
> java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)  
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)  
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)  
> at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)  
> at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)  
> at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)  
> at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)  
> at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)  
> at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)  
> at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)  
> at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)  
> at com.atlassian.confluence.themes.DefaultThemeManager.getThemeSettings(DefaultThemeManager.java:73)  
> at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalThemeKey(DefaultThemeManager.java:49)  
> at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalTheme(DefaultThemeManager.java:42)  
> at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:80)  
> at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:53)  
> at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
> at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:32)  
> at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
> at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)  
> at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:64)  
> at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)  
> at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39)  
> at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)  
> at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
> at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)  
> at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)  
> at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)  
> at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)  
> at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)  
> at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)  
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)  
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)  
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)  
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)  
> at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)  
> at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)  
> at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770)  
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)  
> at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
> at java.lang.Thread.run(Thread.java:748)  
> Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
> at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
> at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)  
> at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
> at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:344)  
> at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313)  
> at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:42)  
> at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)  
> at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)  
> at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)  
> at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)  
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
> at com.sun.proxy.$Proxy55.retrieve(Unknown Source)  
> at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)  
> at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)  
> at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)  
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)  
> ... 70 more
>
> ```
> 018-11-13 23:06:47,676 ERROR [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z4jp4wl1yl4557|6c2fa18e]-HelperThread-#0] [org.postgresql.Driver] connect Connection error: 
> org.postgresql.util.PSQLException: FATAL: the database system is starting up
> ?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
> ?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
> ?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
> ?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
> ?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
> ?? ?at org.postgresql.Driver.connect(Driver.java:252)
> ?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
> ?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
> ?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
> ?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
> ?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
> 2018-11-13 23:06:47,754 WARN [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z4jp4wl1yl4557|6c2fa18e]-HelperThread-#0] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@72131583 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
> org.postgresql.util.PSQLException: FATAL: the database system is starting up
> ?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
> ?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
> ?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
> ?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
> ?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
> ?? ?at org.postgresql.Driver.connect(Driver.java:252)
> ?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
> ?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
> ?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
> ?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
> ?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
> ?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
> ```


author: Network admin
body: 018-11-13 23:06:47,676 ERROR \[C3P0PooledConnectionPoolManager\[identityToken-\>2rvxtv9z4jp4wl1yl4557\|6c2fa18e\]-HelperThread-#0\] \[org.postgresql.Driver\] connect Connection error:  
org.postgresql.util.PSQLException: FATAL: the database system is starting up  
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)  
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)  
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)  
at org.postgresql.jdbc.PgConnection.\<init\>(PgConnection.java:194)  
at org.postgresql.Driver.makeConnection(Driver.java:450)  
at org.postgresql.Driver.connect(Driver.java:252)  
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)  
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)  
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)  
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)  
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)  
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)  
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)  
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)  
2018-11-13 23:06:47,754 WARN \[C3P0PooledConnectionPoolManager\[identityToken-\>2rvxtv9z4jp4wl1yl4557\|6c2fa18e\]-HelperThread-#0\] \[mchange.v2.resourcepool.BasicResourcePool\] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@72131583 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:  
org.postgresql.util.PSQLException: FATAL: the database system is starting up  
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)  
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)  
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)  
at org.postgresql.jdbc.PgConnection.\<init\>(PgConnection.java:194)  
at org.postgresql.Driver.makeConnection(Driver.java:450)  
at org.postgresql.Driver.connect(Driver.java:252)  
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)  
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)  
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)  
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)  
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)  
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)  
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)  
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

14-Nov-2018 00:12:37.386 SEVERE \[http-nio-8090-exec-10\] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage\[errorCode=404, location=/fourohfour.action\]  
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)  
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)  
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)  
at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)  
at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)  
at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)  
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)  
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)  
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)  
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)  
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)  
at com.atlassian.confluence.themes.DefaultThemeManager.getThemeSettings(DefaultThemeManager.java:73)  
at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalThemeKey(DefaultThemeManager.java:49)  
at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalTheme(DefaultThemeManager.java:42)  
at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:80)  
at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:53)  
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:32)  
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)  
at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:64)  
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)  
at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39)  
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)  
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)  
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)  
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)  
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)  
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)  
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)  
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)  
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)  
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)  
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)  
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)  
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)  
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770)  
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)  
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.lang.Thread.run(Thread.java:748)  
Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)  
at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:344)  
at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313)  
at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:42)  
at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)  
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)  
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)  
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy55.retrieve(Unknown Source)  
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)  
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)  
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)  
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)  
... 70 more


author: Network admin
body: 14-Nov-2018 00:12:37.386 SEVERE \[http-nio-8090-exec-10\] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage\[errorCode=404, location=/fourohfour.action\]  
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)  
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)  
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)  
at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)  
at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)  
at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)  
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)  
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)  
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)  
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)  
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)  
at com.atlassian.confluence.themes.DefaultThemeManager.getThemeSettings(DefaultThemeManager.java:73)  
at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalThemeKey(DefaultThemeManager.java:49)  
at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalTheme(DefaultThemeManager.java:42)  
at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:80)  
at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:53)  
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:32)  
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)  
at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)  
at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:64)  
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)  
at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39)  
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)  
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)  
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)  
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)  
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)  
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)  
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394)  
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253)  
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)  
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)  
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)  
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)  
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)  
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)  
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770)  
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)  
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.lang.Thread.run(Thread.java:748)  
Caused by: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted \[public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException\] is not allowed.  
at com.atlassian.confluence.tenant.TenantGate$1.lambda$create$0(TenantGate.java:43)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)  
at com.sun.proxy.$Proxy54.getCurrentSession(Unknown Source)  
at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:344)  
at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313)  
at com.atlassian.confluence.setup.bandana.persistence.dao.hibernate.HibernateConfluenceBandanaRecordDao.getRecord(HibernateConfluenceBandanaRecordDao.java:42)  
at com.atlassian.confluence.setup.bandana.ConfluenceDaoBandanaPersister.retrieve(ConfluenceDaoBandanaPersister.java:29)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)  
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)  
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)  
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy55.retrieve(Unknown Source)  
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)  
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$11(SynchronousExternalCache.java:239)  
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.lambda$foldResult$0(CompletionStageUtils.java:19)  
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)  
... 70 more


author: Network admin
body: I edited the ticket to include the catalina.2018-11-14.log (1st block code)

and the atlassian-confluence.log (2nd block code)


author: Daniel Eads
body: Alright, so I see two errors in your logs:

```
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
```

We see this one fairly regularly, usually on the first time someone tries to start Confluence after a fresh installation. The fix is usually [fixing the DNS resolution of postgres](https://confluence.atlassian.com/confkb/confluence-generates-confluence-is-vacant-error-on-install-779164449.html) on the host where Confluence is running. I generally wouldn't expect this to be what's happening to you over time, unless for some reason the host Confluence is running on intermittently can't get responses back from your DNS server and just dumps its records after their TTL expires. So it is worth testing the diagnosis in this article if you can catch it happening. Or maybe even run a bash script that checks DNS resolution on the Confluence server every minute or two, so you can correlate any Confluence breakages with DNS resolution failure.

The second error:

```
Connection error:
org.postgresql.util.PSQLException: FATAL: the database system is starting up
```

Looks like this can happen in Postgres when trying to connect to a slave database when replication is running. Information I was able to find on this suggested making sure the following line was enabled in **postgresql.conf** (for the replication copy):  

```
hot_standby = on
```

I'm not sure from your screenshot if you've got a hot backup running, so that also seems worth investigating.

Also take a look to make sure [validation queries are enabled](https://confluence.atlassian.com/doc/surviving-database-connection-closures-297665664.html) (these are only enabled by default on fresh installations after Confluence 6.5 - if you've upgraded from an older version, you have to enable it manually). This will get you past tiny disconnects from the database connector.


author: Network admin
body: **On postgresql.conf this is what i see**

```
#hot_standby = off
```

should i turn it on? What does that resolve


author: Daniel Eads
body: With the disclaimer that I'm not a Postgresql expert and *have not tested this*, what I was able to find indicated that this was about still being able to run queries when Postgres is in the process of backing up or making a replication copy.

You can read about this in the [actual Postgresql documentation](https://www.postgresql.org/docs/9.0/hot-standby.html).

Enabling this flag should allow read-only queries to happen. Again, I don't know enough about your database environment to know if you've got Confluence trying to connect to a postgresql slave database. That definitely could be a problem, and if you've got replication enabled (or aren't sure if you don't have it enabled) this would be something you want to track down. If you've got an in-house Postgresql administrator, this would be a good opportunity to buy them lunch.

Cheers,  
Daniel


author: Network admin
body: I am novice to this so please forgive me if my questions seem very obvious to answer by admin standards. I am frustrated because this has been running along side JIRA for 13 months and im just now receiving these errors.

This is the logs by Postgresql

```
2018-11-16 08:49:21 EST LOG:? could not receive data from client: An existing connection was forcibly closed by the remote host.

?? ?
2018-11-16 08:49:23 EST LOG:? received fast shutdown request
2018-11-16 08:49:23 EST LOG:? aborting any active transactions
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST FATAL:? terminating connection due to administrator command
2018-11-16 08:49:23 EST LOG:? autovacuum launcher shutting down
2018-11-16 08:49:24 EST ERROR:? canceling statement due to user request
2018-11-16 08:49:24 EST LOG:? autovacuum launcher shutting down
2018-11-16 08:49:38 EST LOG:? shutting down
2018-11-16 08:49:39 EST LOG:? database system is shut down
```

Here are the latest errors from Confluence

```
2018-11-16 08:32:19,137 ERROR [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z7yvj7j1c59slr|647f8327]-HelperThread-#1] [org.postgresql.Driver] connect Connection error: 
org.postgresql.util.PSQLException: FATAL: the database system is starting up
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
?? ?at org.postgresql.Driver.connect(Driver.java:252)
?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
2018-11-16 08:32:19,184 WARN [C3P0PooledConnectionPoolManager[identityToken->2rvxtv9z7yvj7j1c59slr|647f8327]-HelperThread-#1] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@58e66429 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: the database system is starting up
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:438)
?? ?at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
?? ?at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
?? ?at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
?? ?at org.postgresql.Driver.makeConnection(Driver.java:450)
?? ?at org.postgresql.Driver.connect(Driver.java:252)
?? ?at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
?? ?at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
?? ?at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
?? ?at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
?? ?at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
```


Tags: confluence-server

server