Upgraded 7.7.3 to 7.13.0 - confluence doesn't start
Asked by tgix-msa:

Just updated our installation from 7.7.3 to 7.13.0 and confluence no longer completes the startup procedure.

After waiting several minutes, the java process stops consuming CPU and just sits idle. I have waited 15 minutes, but still not possible to login or any new messages in log. I have tried with the stock server.xml as well.

Running on debian with mysql and java 1.8.0_121 (same as 7.7.3 was running fine on).

Where should I start looking for a solution?

catalina.out after starting 7.13.0:

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED  
\[0.000s\]\[warning\]\[gc\] -Xloggc is deprecated. Will use -Xlog:gc:/home/confluence/atlassian/confluence/logs/gc-2021-08-26_11-22-09.log instead.  
26-Aug-2021 11:22:09.597 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server\] failed to set property \[debug\] to \[0\]  
26-Aug-2021 11:22:09.722 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Connector\] failed to set property \[debug\] to \[0\]  
26-Aug-2021 11:22:09.757 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine\] failed to set property \[debug\] to \[0\]  
26-Aug-2021 11:22:09.761 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host\] failed to set property \[debug\] to \[0\]  
26-Aug-2021 11:22:09.839 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context\] failed to set property \[debug\] to \[0\]  
26-Aug-2021 11:22:09.885 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context\] failed to set property \[debug\] to \[0\]  
26-Aug-2021 11:22:10.510 INFO \[main\] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler \["http-nio-8090"\]  
26-Aug-2021 11:22:10.566 INFO \[main\] org.apache.catalina.startup.Catalina.load Server initialization in \[1144\] milliseconds  
26-Aug-2021 11:22:10.575 INFO \[main\] org.apache.catalina.core.StandardService.startInternal Starting service \[Tomcat-Standalone\]  
26-Aug-2021 11:22:10.577 INFO \[main\] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: \[Apache Tomcat/9.0.45\]  
26-Aug-2021 11:22:13.163 INFO \[Catalina-utility-2\] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.  
log4j:ERROR Could not find value for key log4j.appender.luceneQuery  
log4j:ERROR Could not instantiate appender named "luceneQuery".  
26-Aug-2021 11:22:13.540 INFO \[Catalina-utility-2\] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath  
26-Aug-2021 11:22:13.966 INFO \[Catalina-utility-2\] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'  
11:22:13 \[Catalina-utility-2\] o.a.c.c.C.\[.\[.\[/synchrony-proxy\]\[INFO\] log - Initializing Spring DispatcherServlet 'dispatcher'  
2021-08-26 11:22:14,561 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 7.13.0 \[build 8703 based on commit hash 791df1c41ad1e432a490f9fc6f60e3ea49358bee\] - synchrony version 4.0.0-master-f680ea21  
2021-08-26 11:22:19,803 WARN \[Catalina-utility-1\] \[confluence.impl.setup.DefaultBootstrapDatabaseAccessor\] readValue Unable to determine ZDU status from database. If you are upgrading from a Confluence version prior to 7.9, this is expected.  
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluencedb.CONFZDU' doesn't exist  
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)  
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)  
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)  
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)  
at com.mysql.jdbc.Util.getInstance(Util.java:408)  
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:943)  
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970)  
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906)  
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)  
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677)  
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)  
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2503)  
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1369)  
at com.atlassian.confluence.impl.setup.DefaultBootstrapDatabaseAccessor.readValue(DefaultBootstrapDatabaseAccessor.java:79)  
at com.atlassian.confluence.impl.setup.DefaultBootstrapDatabaseAccessor.getBootstrapData(DefaultBootstrapDatabaseAccessor.java:48)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:805)  
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:191)  
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:144)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)  
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)  
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172)  
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)  
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)  
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.base/java.lang.Thread.run(Unknown Source)  
WARNING: An illegal reflective access operation has occurred  
WARNING: Illegal reflective access by com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy (file:/home/confluence/atlassian/confluence/confluence/WEB-INF/lib/hibernate.adapter-1.0.3.jar) to field java.lang.reflect.Field.modifiers  
WARNING: Please consider reporting this to the maintainers of com.atlassian.hibernate.adapter.proxy.BytecodeProviderImpl_ImplementV2Proxy  
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations  
WARNING: All illegal access operations will be denied in a future release  
Security framework of XStream not explicitly initialized, using predefined black list on your own risk.


Answers:
author: tgix-msa
body: Let it sit overnight and now found the instance at least started:

```
27-Aug-2021 03:26:37.565 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
27-Aug-2021 03:26:37.665 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [57867097] milliseconds
03:26:38 [http-nio-8090-exec-11] o.e.j.u.log[INFO] initialized - Logging initialized @57869906ms to org.eclipse.jetty.util.log.Slf4jLog
```

Errors logged on the webpage when trying to access:

**Pre-Upgrade recovery file generation failed.**   
**Please refer to <https://confluence.atlassian.com/x/ropKGQ> for possible solution.**

I will try the instructions there and disable backup file generation. Disk space is not an issue.

comments: author: tgix-msa
body: Got everything back up again.


author: Martin Henning
body: Letting it sit worked. I left mine for about an hour and it came up


Tags: confluence-server

server