Confluence stopps working after setup and first restart due to memory issues
Asked by Schmeil:

Hi!

I did install confluence server using atlassian-confluence-7.10.2-x64.bin on a Linux Debian Buster with postgresql 10 data base.

I performed the setup without any issues including connecting to the data base, creating an admin user and set up a test space, then I was able to use confluence. But after stopping and starting confluence via /etc/init.d/confluence1 start it doesnt come up again. After some minutes high cpu usage and using \>2gb RAM confluence is stopping by exception.

The logs show "java.lang.OutOfMemoryError", some experiments with variables (eg. -Xms300m -Xmx300m and -XX:ReservedCodeCacheSize=2048m) did not help.

Why the hell confluence stops working after instaling???

The machine has 16GB RAM and 300GB SSD storage, with debian linux 10


Answers:
author: Thiago Masutti
body: Hi [@Schmeil](/t5/user/viewprofilepage/user-id/4233447)

Although the JVM is throwing an Out of Memory error, it seems it is not complaining about the heap space, so changing Xmx and Xms values may not be helpful in this case.

Sometimes the JVM can throw this error when it is unable to create new threads if the OS user has reached the maximum number of processes.

Would you mind login in the OS terminal as the user running the Confluence processes and sharing the output of the following command?

```
ulimit -a
```

If you are running Confluence as a service in Linux, it might be the case where the service is constrained to a certain number of processes/tasks.

Take a look at [this SuSE KB article](https://www.suse.com/support/kb/doc/?id=000015901) on changing the maximum number of tasks/threads/processes allowed by a service.

Let us know if this helps.

Kind regards,  
Thiago

comments: author: Schmeil
body: for the confluence user the output is the following:

$ ulimit -a  
time(seconds) unlimited  
file(blocks) unlimited  
data(kbytes) unlimited  
stack(kbytes) 8192  
coredump(blocks) 0  
memory(kbytes) unlimited  
locked memory(kbytes) 65536  
process 62987  
nofiles 1024  
vmemory(kbytes) unlimited  
locks unlimited  
rtprio 0


author: Thiago Masutti
body: Can you send the Confluence service definition as well?


author: Schmeil
body: Hi [@Thiago Masutti](/t5/user/viewprofilepage/user-id/1209618) , what do you mean by this exactly ?


author: Thiago Masutti
body: If you are using [init.d](https://www.geeksforgeeks.org/what-is-init-d-in-linux-service-management/), there will be a file /etc/init.d/\<*service name*\>.

If you are using [systemd](https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files), there will a .conf file within the /etc/systemd/system/\<*service name*\>.service.d/ folder.

Either of these two will be your service definition.


author: Schmeil
body: #!/bin/bash

### BEGIN INIT INFO  
# Provides: confluence  
# Required-Start: $remote_fs $syslog $time $named  
# Required-Stop: $remote_fs $syslog $time $named  
# Default-Start: 2 3 4 5  
# Default-Stop: 0 1 6  
# Short-Description: Atlassian Confluence  
# Description: Atlassian Confluence Server  
### END INIT INFO

# Confluence Linux service controller script  
cd "/srv/confluence/bin"

case "$1" in  
start)  
./start-confluence.sh  
;;  
stop)  
./stop-confluence.sh  
;;  
restart)  
./stop-confluence.sh  
./start-confluence.sh  
;;  
\*)  
echo "Usage: $0 {start\|stop\|restart}"  
exit 1  
;;  
esac


author: Thiago Masutti
body: Thanks.

So your service definition is standard.

The number of processes allowed to the user seems fine.

Now I'm concerned with the number of open files.

As root, edit the /etc/security/limits.conf file and add the following entries:

```
<os_user_running_confluence> soft nofile 32768
<os_user_running_confluence> hard nofile 32768
```

Try to start the Confluence service and let us know if the same error occurs.

Regards,  
Thiago


author: Schmeil
body: **SOLVED**! Thank you!

**For resolution I performed those steps - maybe one of the solved the problem already:**

* confluence soft nofile 32000  
  confluence hard nofile 64000
* correction of the postgresDb (dropped and recreated)
  * was: Collate C and ctype C (due to wrong suggestion from JIRA installer doc)
  * new: all UTF8
* vim /etc/systemd/system.conf  
  \[Manager\]  
  DefaultTasksMax=15288
* reboot

**btw. I did reset the OPTIONS to default values:**

/srv/confluence/jre//bin/java -Djava.util.logging.config.file=/srv/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options= -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx1024m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/srv/confluence/logs/gc-2021-01-22_00-49-37.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/srv/confluence/logs/gc-2021-01-22_00-49-37.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions -Dignore.endorsed.dirs= -classpath /srv/confluence/bin/bootstrap.jar:/srv/confluence/bin/tomcat-juli.jar -Dcatalina.base=/srv/confluence -Dcatalina.home=/srv/confluence -Djava.io.tmpdir=/srv/confluence/temp org.apache.catalina.startup.Bootstrap start

**Consulted docs:**

* <https://docs.oracle.com/en/java/javase/11/tools/java.html#GUID-3B1CE181-CD30-4178-9602-230B800D4FAE>
* <https://stackoverflow.com/questions/5090858/how-do-you-change-the-character-encoding-of-a-postgres-database>
* <https://stackoverflow.com/questions/12079087/fork-retry-resource-temporarily-unavailable>
* <https://www.thegeekdiary.com/understanding-etc-security-limits-conf-file-to-set-ulimit/>


author: Nic Brough -Adaptavist-
body: Where are you setting the memory variables? Actually, only the "max heap memory" one matters much here - the others you've mentioned have performance impacts and could throw other errors, but they won't lead to "out of memory".

-Xmx300m is the right format (I mention this because some people seem to think it is case insensitive - it's not), and although 300m is too low for Confluence, your next test with 8000m should have worked, which makes me suspect that your numbers might simply not be being applied.

When you get the error, could you look back up through the log to the point where it started? Confluence echoes out the memory settings it has ended up running with and my guess is that it's not inherited the one you have set for some reason.

comments: author: Schmeil
body: Hi, as far I know Xms is minimum and Xmx maximum heap size. Thats why I tested it with this set parameters - unfortunately without a solution :

Using CATALINA_OPTS: -XX:ReservedCodeCacheSize=2000m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options='' -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx8000m -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/srv/confluence/logs/gc-2021-01-22_11-44-25.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/srv/confluence/logs/gc-2021-01-22_11-44-25.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions

Could it be that there are other java / memory parameters needs to be changed on the machine?


author: Nic Brough -Adaptavist-
body: What does the application log say the parameters are coming out as when you start the applicaiton?


author: brbojorque
body: Hi [@Schmeil](/t5/user/viewprofilepage/user-id/4233447) ,

300mb memory does not cut it in Confluence. Try these variables

```
-Xms8000m -Xmx8000m
```

And update this variable XX:ReservedCodeCacheSize to the default value whatever it is.

comments: author: Schmeil
body: thank you.. unfortunately those parameters doesnt help.. I still see such log messages after starting:

\[93,640s\]\[warning\]\[os,thread\] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.Exception in thread "Navlink Plugin Executor:thread-2" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang.Thread.start0(Native Method) at java.base/java.lang.Thread.start(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)


author: brbojorque
body: Hi [@Schmeil](/t5/user/viewprofilepage/user-id/4233447) ,  
May I know the full path of the file you've updated the JVM values with?

Check also the available memory by typing \`free -h\`


author: Schmeil
body: Hi, the file is: /srv/confluence/bin/setenv.sh.

Changes in this file are effectively, e.g. trying to set the "JVM_MAXIMUM_MEMORY" variable (which can be set in JIRA) leads to error messages in the log.

Starting confluence with used 700MB RAM (of 16GB max.) on the machine ends with used 3,5 GB used after 1,5 minutes .. at this time lot of error messages like this start appearing:

\[95,394s\]\[warning\]\[os,thread\] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.

Also this message is displayd:

Exception in thread "Catalina-utility-2" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

**free -h says:**

total used free shared buff/cache available  
Mem: 16Gi 3,5Gi 10Gi 69Mi 2,2Gi 12Gi


Tags: confluence-server

server