Confluence Fails to load after DB Update
Asked by Andy:

I am running CentOS 7, and uninstalled my old version of postgresql after backing my db up. I am now running version 11, and I did a restore of the db. When I attempt to visit confluence in my browser, this is what I am seeing.

<https://ibb.co/XkFFMP0>

Update

Refreshing the page now shows this:

HTTP Status 500 -- Internal Server Error {#toc-hId--1242572317}
---------------------------------------------------------------

*** ** * ** ***

**Type** Exception Report

**Message** java.io.IOException: java.lang.reflect.InvocationTargetException

**Description** The server encountered an unexpected condition that prevented it from fulfilling the request.

**Exception**

```
java.lang.IllegalStateException: java.io.IOException: java.lang.reflect.InvocationTargetException	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:101)	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
```

**Root Cause**

```
java.io.IOException: java.lang.reflect.InvocationTargetException	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:236)	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
```

**Root Cause**

```
java.lang.reflect.InvocationTargetException	java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)	java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)	java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233)	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
```

**Root Cause**

```
java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)	java.base/java.io.RandomAccessFile.open0(Native Method)	java.base/java.io.RandomAccessFile.open(Unknown Source)	java.base/java.io.RandomAccessFile.<init>(Unknown Source)	java.base/java.io.RandomAccessFile.<init>(Unknown Source)	java.base/java.util.zip.ZipFile$Source.<init>(Unknown Source)	java.base/java.util.zip.ZipFile$Source.get(Unknown Source)	java.base/java.util.zip.ZipFile$CleanableResource.<init>(Unknown Source)	java.base/java.util.zip.ZipFile$CleanableResource.get(Unknown Source)	java.base/java.util.zip.ZipFile.<init>(Unknown Source)	java.base/java.util.zip.ZipFile.<init>(Unknown Source)	java.base/java.util.jar.JarFile.<init>(Unknown Source)	java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)	java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)	java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)	org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:233)	org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:162)	org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:307)	org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:97)	org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)	org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:282)	org.apache.catalina.webresources.Cache.getResource(Cache.java:62)	org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:217)	org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:226)	org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2303)	org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1334)	org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1188)	com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:38)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:59)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
```

**Note** The full stack trace of the root cause is available in the server logs.

*** ** * ** ***

### Apache Tomcat/9.0.33 {#toc-hId--552010843}

I did run the initdb command.


Answers:
author: Nic Brough -Adaptavist-
body: Ok, the core of the problem is "Jira cannot connect to the database", but you've given us enough information to track that down to why - it can't actually open the database driver.

```
java.io.FileNotFoundException: /opt/atlassian/confluence/confluence/WEB-INF/lib/atlassian-http-2.0.8.jar (Too many open files)
```

is not 100% clear - what it says is "file not found", but what it means is "I can't open a file", not that the file is not there.

The "too many open files" is the culprit, that means the OS is limited to only opening a certain number of files (by the current user) and you have run into that limit. It is a bit archaic, one of those things that really should just go away as it is mostly pointless as hardware and software have evolved way past it (a bit like Windows for example)

Anyway, the history of it is meh. There's a fix written up at <https://confluence.atlassian.com/confkb/too-many-open-files-error-659784628.html> - it boils down to "change the ulimit for the user and restart the application"


Tags: confluence-server

server