Confluence Server in Docker fails to start after accidential Upgrade
Asked by Olaf Koester:

Hello,

I have a problem with my Confluence server in docker container.

I'm using atlassian/confluence-server

Yesterday I changed something within my Confluence instance running 6.4.X (I dont remember X, but sure it was 6.4.whatever release).

I wanted my Instance to be accessible from Internet behind reverse proxy. So I changed base URL to the external DNS.

Then I got an popup regarding Confluence unable to verify base URL because of proxy settings maybe wrong.

To fix this, I stopped docker container with

```
docker stop confluence
```

and removed it with

```
docker rm confluence
```

and since this is my only install, I removed the image:

```
docker rmi atlassian/confluence-server
```

Then I wanted to restart the container with this line:

```
run --restart always -v /srv/docker/confluence:/var/atlassian/application-data/confluence -e "CATALINA_OPTS=-Xms512m -Xmx768m" -e "CATALINA_CONNECTOR_PROXYNAME=wiki.digitales.land" -e "CATALINA_CONNECTOR_PROXYPORT=443" -e "CATALINA_CONNECTOR_SCHEME=https" -e "CATALINA_CONNECTOR_SECURE=true" --memory=4G --dns=8.8.8.8 --name="confluence" -d -p 8090:8090 atlassian/confluence-server
```

(Note that String "CATALINA_OPTS=..." was in the command line before, only proxy related settings were added)

After that command (and waiting for some minutes), I tried to log in again into Confluence web login but saw this error:
>
> HTTP Status 500 - java.lang.IllegalStateException: Spring Application context has not been set {#toc-hId-1764884989}
> --------------------------------------------------------------------------------------------------------------------
>
> **type** Exception report
>
> **message** ++java.lang.IllegalStateException: Spring Application context has not been set++
>
> **description** ++The server encountered an internal error that prevented it from fulfilling this request.++
>
> **exception**
>
> ```
> com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Spring Application context has not been set	com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)	com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)	com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20)	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62)	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:31)	com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
> ```
>
> **root cause**
>
> ```
> java.lang.IllegalStateException: Spring Application context has not been set	com.atlassian.spring.container.SpringContainerContext.getComponent(SpringContainerContext.java:48)	com.atlassian.spring.container.ContainerManager.getComponent(ContainerManager.java:33)	com.atlassian.confluence.util.LazyComponentReference$Accessor.get(LazyComponentReference.java:42)	com.atlassian.util.concurrent.Lazy$Strong.create(Lazy.java:85)	com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)	com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)	com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)	com.atlassian.confluence.plugin.servlet.filter.ServletFilterModuleContainerFilter.getServletModuleManager(ServletFilterModuleContainerFilter.java:20)	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:62)	com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)	com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)	com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:72)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:49)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:65)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:54)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:31)	com.atlassian.confluence.impl.servlet.HoldingUntilStartedFilter.doFilter(HoldingUntilStartedFilter.java:88)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:44)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
> ```
>
> **note** ++The full stack trace of the root cause is available in the Apache Tomcat/8.0.47 logs.++

I noticed that I've accidentially tried to upgrade Confluence to 6.5 because of inherit pulling a new image...

So I was looking for several hours to fix that.

When looking into

```
docker logs confluence
```

this messages came up:
> User is currently root. Will change directory ownership to daemon:daemon, then downgrade permission to daemon  
> executing as current user  
> If you encounter issues starting up Confluence, please see the Installation guide at <http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide>  
>
> Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out  
> ---------------------------------------------------------------------------  
> Using Java: /usr/lib/jvm/java-1.8-openjdk/bin/java  
> 2017-11-09 08:54:51,397 INFO \[main\] \[atlassian.confluence.bootstrap.SynchronyProxyWatchdog\] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required  
> ---------------------------------------------------------------------------  
> 09-Nov-2017 08:54:51.794 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin \[SetPropertiesRule\]{Server} Setting property 'debug' to '0' did not find a matching property.  
> 09-Nov-2017 08:54:51.859 WARNING \[main\] org.apache.catalina.startup.SetAllPropertiesRule.begin \[SetAllPropertiesRule\]{Server/Service/Connector} Setting property 'proxyPort' to '' did not find a matching property.  
> 09-Nov-2017 08:54:51.861 WARNING \[main\] org.apache.catalina.startup.SetAllPropertiesRule.begin \[SetAllPropertiesRule\]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.  
> 09-Nov-2017 08:54:51.870 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin \[SetPropertiesRule\]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.  
> 09-Nov-2017 08:54:51.876 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin \[SetPropertiesRule\]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.  
> 09-Nov-2017 08:54:51.912 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin \[SetPropertiesRule\]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.  
> 09-Nov-2017 08:54:51.934 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin \[SetPropertiesRule\]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.  
> 09-Nov-2017 08:54:52.211 INFO \[main\] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler \["http-nio-8090"\]  
> 09-Nov-2017 08:54:52.230 INFO \[main\] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read  
> 09-Nov-2017 08:54:52.233 INFO \[main\] org.apache.catalina.startup.Catalina.load Initialization processed in 535 ms  
> 09-Nov-2017 08:54:52.241 INFO \[main\] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone  
> 09-Nov-2017 08:54:52.241 INFO \[main\] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.47  
> 09-Nov-2017 08:54:54.048 INFO \[localhost-startStop-2\] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: \[com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@294214c8\]  
> 09-Nov-2017 08:54:54.326 INFO \[localhost-startStop-2\] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'  
> 09-Nov-2017 08:54:54.326 INFO \[localhost-startStop-2\] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started  
> 09-Nov-2017 08:54:54.334 INFO \[localhost-startStop-2\] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date \[Thu Nov 09 08:54:54 GMT 2017\]; root of context hierarchy  
> 09-Nov-2017 08:54:54.430 INFO \[localhost-startStop-2\] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: \[class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig\]  
> 2017-11-09 08:54:54,440 INFO \[localhost-startStop-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 6.5.0 \[build 7502 based on commit hash b9217199c27dd617c8d51f6186067d7767c5001b\] - synchrony version 2.1.0-release-confluence_6.5-8d720dbe  
> 09-Nov-2017 08:54:54.959 INFO \[localhost-startStop-2\] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'  
> 09-Nov-2017 08:54:55.005 INFO \[localhost-startStop-2\] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path \[/v1/bayeux-sync1\] onto handler of type \[class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler\]  
> 09-Nov-2017 08:54:55.243 INFO \[localhost-startStop-2\] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{\[/reload\],methods=\[PUT\]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)  
> 09-Nov-2017 08:54:55.245 INFO \[localhost-startStop-2\] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{\[/healthcheck\]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()  
> 09-Nov-2017 08:54:55.566 INFO \[localhost-startStop-2\] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date \[Thu Nov 09 08:54:54 GMT 2017\]; root of context hierarchy  
> 09-Nov-2017 08:54:55.623 INFO \[localhost-startStop-2\] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path \[/\*\*\] onto handler of type \[class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler\]  
> 09-Nov-2017 08:54:55.725 INFO \[localhost-startStop-2\] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647  
> 09-Nov-2017 08:54:55.779 INFO \[localhost-startStop-2\] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1451 ms  
> 2017-11-09 08:54:57,740 INFO \[localhost-startStop-1\] \[atlassian.confluence.cluster.DefaultClusterConfigurationHelper\] lambda$populateExistingClusterSetupConfig$1 Populating setup configuration if running with Cluster mode...  
> **2017-11-09 08:54:57,878 ERROR \[localhost-startStop-1\] \[atlassian.confluence.setup.ConfluenceConfigurationListener\] contextInitialized An error was encountered while bootstrapping Confluence (see below):**   
> **Unable to bootstrap application: The database driver (com.mysql.jdbc.Driver) was not found in the class path.\<br/\>This Confluence installation cannot be upgraded automatically. Please see Confluence documentation:\<br/\><https://confluence.atlassian.com/display/DOC/Database+Setup+For+MySQL>**   
> **com.atlassian.config.bootstrap.BootstrapException: Unable to bootstrap application: The database driver (com.mysql.jdbc.Driver) was not found in the class path.\<br/\>This Confluence installation cannot be upgraded automatically. Please see Confluence documentation:\<br/\><https://confluence.atlassian.com/display/DOC/Database+Setup+For+MySQL>**   
> at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:39)  
> at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:127)  
> at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:60)  
> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)  
> at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)  
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)  
> at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)  
> at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
> at java.lang.Thread.run(Thread.java:745)  
> 2017-11-09 08:54:57,895 WARN \[localhost-startStop-1\] \[atlassian.confluence.plugin.PluginFrameworkContextListener\] contextInitialized Not starting full plugin system due to upgrade or licensing errors  
> 2017-11-09 08:54:58,610 INFO \[localhost-startStop-1\] \[com.atlassian.confluence.lifecycle\] init Confluence is ready to serve  
> 09-Nov-2017 08:54:58.635 INFO \[main\] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler \["http-nio-8090"\]  
> 09-Nov-2017 08:54:58.644 INFO \[main\] org.apache.catalina.startup.Catalina.start Server startup in 6411 ms

Note the bold lines. I double checked that the MySQL driver is in the path ./WEB-INF/lib/ as it was already before the failure.

To be sure, I also downloaded a new driver and copied the jar to that folder. Error persists....

What I tried also:

<https://community.atlassian.com/t5/Confluence-questions/Confluence-Server-install-in-Docker-falure/qaq-p/458625>

and followed this:

<https://community.atlassian.com/t5/Confluence-questions/Confluence-not-starting-after-instalation/qaq-p/604520>

besides other things like checking permissions on persistent folder and some more. Nothing worked...

Another information aside:

When that happened yesterday, I had an invalid license in place. This I already renewed today and also changed it inside confluence.cfg.xml

I really need some support on this.

Does someone have a suggestion?

What will happen, if I'd remove confluence.cfg.xml to trigger a new installation? Will all data inside persistent folder and database be erased? Is it possible to do so and bring the last backup (database and files) back in place to get working installation back?

Really looking forward to your suggestions.

Regards

Olaf


Answers:
author: thesefer
body: As far as I have seen 6.5 requires oracle jdk but the Dockerfile/Container provided uses openjdk which is not supported.

You would have to build it yourself but this is not possible as described at
> <https://confluence.atlassian.com/confkb/update-the-confluence-docker-image-to-use-oracle-jdk-829062521.html>

because the source doesn't exists

```
Step 1/20 : FROM oracle/JDK:8
invalid reference format: repository name must be lowercase

**alternative try**

Step 1/20 : FROM oracle/jdk:8
pull access denied for oracle/jdk, repository does not exist or may require 'docker login'?
```

You would need to modify the Dockerfile to download and provide the mysql-driver, too.

comments: author: Olaf Koester
body: Hello S K,

thanks for your response.

It turned out (after hours of debugging) that really only the mysql driver was the problem.

I followed the Howto for using MySQL driver and placed it into \<INSTALL-DIR\>/confluence/WEB-INF/lib/ as recommended (better: it was already there from before).

Didn't work.

So I decided to give a fresh "reinstall" with actual docker image atlassian/confluence-server:latest, an empty persistant directory and an empty database.

Then, in the Database Setup Dialog I chose MySQL, but were not able to continue. Confluence stated that it was missing MySQL connector.

I created the path mentioned above ("\*/lib/\*) and placed the .jar file into it. Checked also for the rigt permissions.

Restart... Nothing... Same problem.

Then I found this inside the configuration screen (browser):

Drop the .jar file in **/opt/atlassian/confluence/confluence/WEB-INF/lib**

This path is completely different from the one mentioned in the Confluence Documentation on web.

So I connected to the running docker container into a shell and copied the .jar connector file into the path mentioned above.

Exited and stopped the docker container.

Commited the changed image:

```
docker commit <container ID> local/confluence-server
```

Last step was to create a new container (see in my very first post) from my new image local/confluence-server

Because my image now contains the MySQL connector in the right place, my new container came up immediately and I was able to login again.

No data got lost!

Note that this can only be a workaround.

I Think that there's an error in the official image. Seems that the information where to find the MySQL connector in the (persistant) path is missing...

Hope my experiences will help others.

Regards

Olaf


author: thesefer
body: Glad to hear you were able to resolve your issue!

But path is the same as in the how to according to your post.
> `<installation-directory>/confluence/WEB-INF/lib`


author: Brad Ash
body: 3 years later and this is still a problem, Docker Hub README says to copy the .jar file into

    /opt/atlassian/confluence/lib

when it really needs to be

    /opt/atlassian/confluence/confluence/WEB_INF/lib

There is no mention of WEB_INF folder anywhere on this page (<https://hub.docker.com/r/atlassian/confluence-server/>), yet according to the 7.x docs it's still the correct location (<https://confluence.atlassian.com/doc/database-setup-for-mysql-128747.html>)


author: Andy Heinzer
body: Hi [@Brad Ash](/t5/user/viewprofilepage/user-id/3903280)

Thanks for calling this out. I made a documentation bug ticket about this over in <https://jira.atlassian.com/browse/SCALE-16>

Sorry we've had the wrong info documented here. We'll get this fixed.

Andy