After Confluence Upgrade One or more Filters failed to start
Asked by d-moore1:

Hello,

I recently upgraded our confluence server from 5.3 to the latest version to deal with one of the recent security vulnurabilities. I believe I've completed the upgrade successfully, but confluence won't start. We're seeing a severe error in catalina.out followed by a number of warnings.

```
23-May-2019 07:06:35.742 INFO [Catalina-utility-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
23-May-2019 07:06:35.763 INFO [Catalina-utility-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
23-May-2019 07:06:36.369 INFO [Catalina-utility-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
23-May-2019 07:06:36.509 INFO [Catalina-utility-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 7567 ms
23-May-2019 07:12:20.484 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
23-May-2019 07:12:20.485 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
23-May-2019 07:12:32.317 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
23-May-2019 07:12:32.318 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
23-May-2019 07:12:32.318 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
23-May-2019 07:12:32.319 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
23-May-2019 07:12:32.319 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
23-May-2019 07:12:32.319 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.hsqldb.jdbc.JDBCDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
23-May-2019 07:12:32.330 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.332 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.335 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.336 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.337 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.338 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.339 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.340 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.341 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.343 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
23-May-2019 07:12:32.344 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [HSQLDB Timer @5fc0d381] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)
org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
java.lang.Thread.run(Thread.java:748)
23-May-2019 07:12:32.345 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AtlassianEvent::CustomizableThreadFactory-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
23-May-2019 07:12:32.397 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio2-8090"]
23-May-2019 07:12:32.402 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [375,062] milliseconds
```

I've verified that there is a java process running under the confluence dedicated user, and that the process is bound to the port we've specified in the server.xml file. However there is no atlassian-confluence.log file in the confluence/logs directory, so I'm not sure how to proceed.

Does anyone have any ideas?

Thanks,  
Dan


Answers:
author: Steven F Behnke
body: there are two directories for logs. the install directory and the home directory. the default installation directory logs is /opt/atlassian/confluence/logs/ and the default home directory logs is /var/atlassian/application-data/confluence/logs.

do you have the home logs directory?

comments: author: d-moore1
body: Thanks Steven. No, unfortunately I only seem to have the installation directory. I have /var/atlassian/application-data which contains my other Atlassian products, but no "confluence" directory.


author: d-moore1
body: Okay I think I figured it out. In case anyone else is missing a Confluence home directory, mine was in /bin/atlassian. I was able to track the problem down to this issue: <https://confluence.atlassian.com/confkb/upgrade-to-confluence-6-or-above-fails-when-using-the-embedded-database-917963158.html>

Still working through other issues, but I was able to make it past this problem.

comments: author: Colin McDermott_GLiNTECH
body: Hello [@d-moore1](/t5/user/viewprofilepage/user-id/1025824)

Just to clarify, that location seems very strange. It could be correct, but /bin/atlassian is very VERY non standard and probably something I would plan to change in future as:

/bin is usually on linux systems reserved for executable programs, i.e. stuff you run.

/bin/atlassian to me points to all \*atlassian\* products. If say both jira and confluence used this directory it would be BAD, as many folders would overlap and they would write over each other.

To find where this location is set, goto:

\<your confluence install directory\*\>/confluence/WEB-INF/classes

You will find a file there called: confluence-init.properties

All the lines with # infront do not get read.

You will see one line called confluence.home=

That is where it is set. It may not be set at all, then Confluence will assume your current directory is where it should start.

Ofcourse your system may be working fine with that home directory above. I suspect what has happened though is on starting confluence it has started in /bin or /bin/atlassian randomly and this has become your home directory.

Now to your problem.

Your upgrade path is broken.

<https://confluence.atlassian.com/doc/upgrading-confluence-4578.html>

Please revert to Confluence 5.3 then upgrade to confluence 5.10.9

From confluence 5.10.9 you should then proceed to the LTS versions 6.13.18 OR 7.4.6

Yes stick with LTS if your client is doing infrequent updates.

\*this is where you got the catalina log above, but one level up.


author: Ross Brook
body: Thanks Colin this answer has helped me with an issue


Tags: confluence-server

server

startup