Issue with upgrading confluence to latest version
Asked by Fluccs Australia:

Hello Support,

We have tried to upgrade directly to the latest version of [Confluence 8](https://www.atlassian.com/software/confluence/download-archives).

and got the error  
\~\~\~  
confluence + Database: The database collation 'utf8_bin' is not supported by Confluence. You need to use 'utf8mb4_bin'.  
So updated the database charset and collation and then tried to upgrade confluence but got the error:  
*Exception in thread "I/O dispatcher 2" Exception in thread "I/O dispatcher 1" Exception in thread "I/O dispatcher 5" java.lang.OutOfMemoryError: Java heap space*  
*Exception in thread "Navlink Plugin Executor:thread-1" java.lang.IllegalMonitorStateException*  
*at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(Unknown Source)*  
*at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)*  
*at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)*  
*at java.base/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)*  
*at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)*  
*at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)*  
*at java.base/java.lang.Thread.run(Unknown Source)*  
*java.lang.OutOfMemoryError: Java heap space*  
*Exception in thread "I/O dispatcher 6" java.lang.OutOfMemoryError: Java heap space*  
*java.lang.OutOfMemoryError: Java heap space*  
*Exception in thread "I/O dispatcher 8" java.lang.OutOfMemoryError: Java heap space*  
*at java.base/java.util.Collections$UnmodifiableCollection.iterator(Unknown Source)*  
*at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:210)*  
*at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)*  
*at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)*  
*at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)*  
*at java.base/java.lang.Thread.run(Unknown Source)*  
*\~\~\~*  
Could you please have a check into this and let us know the work around for this.


Answers:
author: Nic Brough -Adaptavist-
body: Welcome to the Atlassian Community!

It would be useful to know what version of Confluence you started with.

But the error message is pretty clear, the system has not got enough memory to do the work. Confluence's functionality and capabilities have grown over the years, and that means that more recent versions need more memory space to work in.

What are the -Xms and -Xmx settings you have for this server?

And (very roughly) how big is the system? A few thousand pages, tens of thousands, hundreds of thousands? More?

comments: author: Fluccs Australia
body: Hi [@Nic Brough -Adaptavist-](/t5/user/viewprofilepage/user-id/781454)

Thanks for getting back to us

We had an 8GB RAM server and we increased the RAM to 16GB to find out the same problem.

The -Xms and -Xmx settings in *bin/setenv.sh* are currently set to   
CATALINA_OPTS="-Xms1024m -Xmx1024m ${CATALINA_OPTS}"  

However, we did try to increase it to 2GB... should this be increased prior we initiate the upgrade?

There are hundreds of pages yes.

Regards

Andy


author: Nic Brough -Adaptavist-
body: The short answer is "yes".

Your data sizes are not a problem, your server looks like it is more than capable, so I am driven towards the memory allocations.

For smaller and older Confluences, -Xmx1024m is not a bad setting, but as you have increased the server memory so much, you should take advantage of it. Today, I would give Confluence 4Gb to work with as the starting point.


author: Fluccs Australia
body: did as you suggested, and now when we upgrade from 7.18.1 to 7.20

we get an error below

executing using dedicated user  
If you encounter issues starting up Confluence, please see the Installation guide at <http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide>

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out  
---------------------------------------------------------------------------  
Using Java: /opt/atlassian/confluence/jre//bin/java  
2023-01-26 15:11:07,489 INFO \[main\] \[atlassian.confluence.bootstrap.SynchronyProxyWatchdog\] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required  
---------------------------------------------------------------------------  
Using CATALINA_BASE: /opt/atlassian/confluence  
Using CATALINA_HOME: /opt/atlassian/confluence  
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp  
Using JRE_HOME: /opt/atlassian/confluence/jre/  
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar  
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-11-07.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m  
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid  
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED  
Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer  
SEVERE: Could not contact \[localhost:8000\] (base port \[8000\] and offset \[0\]). Tomcat may not be running.  
Jan 26, 2023 3:11:07 PM org.apache.catalina.startup.Catalina stopServer  
SEVERE: Error stopping Catalina  
**java.net.ConnectException: Connection refused (Connection refused)**   
**at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)**   
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)  
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)  
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)  
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)  
at java.base/java.net.Socket.connect(Unknown Source)  
at java.base/java.net.Socket.connect(Unknown Source)  
at java.base/java.net.Socket.\<init\>(Unknown Source)  
at java.base/java.net.Socket.\<init\>(Unknown Source)  
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:667)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)  
at java.base/java.lang.reflect.Method.invoke(Unknown Source)  
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:391)  
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:481)

The stop command failed. Attempting to signal the process to stop through OS signal.  
Tomcat stopped.

--------------------

catalina.out error:  

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED  
\[0.001s\]\[warning\]\[gc\] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-15-26.log instead.  
26-Jan-2023 15:15:27.638 INFO \[main\] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler \["http-nio-8090"\]  
26-Jan-2023 15:15:27.671 INFO \[main\] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler \["http-nio-8090"\]  
**26-Jan-2023 15:15:27.672 SEVERE \[main\] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component \[Connector\[HTTP/1.1-8090\]\]**   
**org.apache.catalina.LifecycleException: Protocol handler initialization failed**   
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)  
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)  
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)  
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)  
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)  
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)  
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)  
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)  
at java.base/java.lang.reflect.Method.invoke(Unknown Source)  
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)  
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)  
Caused by: java.net.BindException: Address already in use  
at java.base/sun.nio.ch.Net.bind0(Native Method)  
at java.base/sun.nio.ch.Net.bind(Unknown Source)  
at java.base/sun.nio.ch.Net.bind(Unknown Source)  
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)  
at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:275)  
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:230)  
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1227)  
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1240)  
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:603)  
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)  
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1048)  

Please help us fix this problem. We have custom entries in our server.xml file

\<Connector port="8090" maxThreads="550"  
minSpareThreads="35"  
connectionTimeout="20000"

enableLookups="false"  
maxHttpHeaderSize="8192"  
protocol="HTTP/1.1"  
useBodyEncodingForURI="true"  
redirectPort="8443"  
acceptCount="150"  
disableUploadTimeout="true"  
scheme="https"  
proxyName="confluence.xxxxxxxxxxx.net"  
proxyPort="443" /\>

\[root@confluence conf\]# netstat -nl \| grep 8090  
tcp6 0 0 :::8090 :::\* LISTEN  
\[root@confluence conf\]#

Regards

Andy


author: Fluccs Australia
body: adding one more error I found when testing the new configuration

\[root@confluence bin\]# ./configtest.sh  
If you encounter issues starting up Confluence, please see the Installation guide at <http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide>

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out  
---------------------------------------------------------------------------  
Using Java: /opt/atlassian/confluence/jre//bin/java  
2023-01-26 15:34:45,215 INFO \[main\] \[atlassian.confluence.bootstrap.SynchronyProxyWatchdog\] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required  
---------------------------------------------------------------------------  
Using CATALINA_BASE: /opt/atlassian/confluence  
Using CATALINA_HOME: /opt/atlassian/confluence  
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp  
Using JRE_HOME: /opt/atlassian/confluence/jre/  
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar  
Using CATALINA_OPTS: -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dconfluence.context.path= -Djava.locale.providers=JRE,SPI,CLDR -Dsynchrony.enable.xhr.fallback=true -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/opt/atlassian/confluence/logs/gc-2023-01-26_15-34-45.log::filecount=5,filesize=2M -XX:G1ReservePercent=20 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=256m -Xms4096m -Xmx4096m  
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid  
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED  
Jan 26, 2023 3:34:45 PM org.apache.catalina.core.StandardContext setPath  
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path \[${confluence.context.path}/synchrony-proxy\] does not meet these criteria and has been changed to \[/${confluence.context.path}/synchrony-proxy\]  
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init  
INFO: Initializing ProtocolHandler \["http-nio-8090"\]  
Jan 26, 2023 3:34:46 PM org.apache.coyote.AbstractProtocol init  
INFO: Initializing ProtocolHandler \["http-nio-8090"\]  
Jan 26, 2023 3:34:46 PM org.apache.catalina.util.LifecycleBase handleSubClassException  
SEVERE: Failed to initialize component \[Connector\[HTTP/1.1-8090\]\]  
org.apache.catalina.LifecycleException: Protocol handler initialization failed  
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1051)  
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)  
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:556)  
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)  
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045)  
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)  
at org.apache.catalina.startup.Catalina.load(Catalina.java:724)  
at org.apache.catalina.startup.Catalina.load(Catalina.java:746)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)  
at java.base/java.lang.reflect.Method.invoke(Unknown Source)  
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)  
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:483)  
**Caused by: java.net.BindException: Address already in use**   
at java.base/sun.nio.ch.Net.bind0(Native Method)  
at java.base/sun.nio.ch.Net.bind(Unknown Source)  
at java.base/sun.nio.ch.Net.bind(Unknown Source)  
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)

- Andy


author: Nic Brough -Adaptavist-
body: Thanks for the bold highlights, it makes it a lot easier to see where you think the problem is without shortening the rest of the data that we might want to ask you later!

The last one jumps out to me immediately. It might be the cause of the other two (I would bet that it is, but I can not be certain).

What it tells us is that there is something else on the server that is running on that port.

When it comes to Atlassian installation problems, this usually means that while the upgrade has worked (or at least got to a point where it's going to start serving web pages), the old version was still running, and hence hogging the port. It might not be obvious that the old one was not shut down in full, and sometimes, you can even get "port is in use already" with a completely dead system.

I see you're using a sensible operating system, so there's a few simple commands you can look at to see what is going on.

Visit Confluence in a browser

You may just get a pile of errors, but the important bit is that the connection on that port responds. If you don't get a "nothing running here" type error, you have something running that you need to find and stop.

* ps -ef
  * Is the most simple one. A list of processes running - is there any reference to any Confluences there? If so, try shutting them down (resort to kill -9 if you have to), and restarting the new install. This is however very very blunt and not the best idea on a production system, it can lead to a bit of data corruption, especially if people are editing pages when you do it.
* netstat -plant / netstat -plank
  * The flags vary a bit on my boxes, but the results should list all processes with their listening ports whatever your OS. So you can target the "kill -9" better.
  * But. Netstat is old, and falling out of use, you may not have it. Don't bother installing it if you've got the next option.
* ss -tupln
  * is the more modern version which will list active ports

In all cases, adding "\| grep 8090" is going to make the output really simple!

Note - in some cases, you need to jump through a lot of hoops to "unlock" a port. Generally, it is far easier to stop the process doing it from starting on reboot, and then reboot the server!

TLDR: find what is blocking the port and kill it.


Tags: confluence-server

server