java.lang.OutOfMemoryError: unable to create new native thread
Asked by Kosta:

Hi,

on startup of Confluence I'm getting a lot of OutOfMemory messages.

Setting CATALINA_OPTS="-Xms2048m -Xmx2048m -XX:+UseG1GC ${CATALINA_OPTS}" didn't solve the problem.

--------------------------

02-Sep-2018 21:15:14.916 INFORMATION \[main\] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler \["http-nio-8090"\]  
02-Sep-2018 21:15:14.999 INFORMATION \[main\] org.apache.catalina.startup.Catalina.start Server startup in 210098 ms  
Exception in thread "Dispatch-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@49a60313" java.lang.OutOfMemoryError: unable to create new native th  
read  
at java.lang.Thread.start0(Native Method)  
at java.lang.Thread.start(Thread.java:717)  
at com.mchange.v2.async.ThreadPerTaskAsynchronousRunner$DispatchThread.run(ThreadPerTaskAsynchronousRunner.java:214)  
Exception in thread "C3P0PooledConnectionPoolManager\[identityToken-\>ntek999x1778udb1lzcqhe\|3dffa4e1\]-AdminTaskTimer" java.lang.OutOfMemoryError: unable to create  
new native thread  
at java.lang.Thread.start0(Native Method)  
at java.lang.Thread.start(Thread.java:717)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner.recreateThreadsAndTasks(ThreadPoolAsynchronousRunner.java:535)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner.access$400(ThreadPoolAsynchronousRunner.java:49)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector.run(ThreadPoolAsynchronousRunner.java:825)  
at java.util.TimerThread.mainLoop(Timer.java:555)  
at java.util.TimerThread.run(Timer.java:505)  
Exception in thread "C3P0PooledConnectionPoolManager\[identityToken-\>ntek999x1778udb1lzcqhe\|71b539b\]-AdminTaskTimer" java.lang.OutOfMemoryError: unable to create new native thread  
at java.lang.Thread.start0(Native Method)  
at java.lang.Thread.start(Thread.java:717)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner.recreateThreadsAndTasks(ThreadPoolAsynchronousRunner.java:535)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner.access$400(ThreadPoolAsynchronousRunner.java:49)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector.run(ThreadPoolAsynchronousRunner.java:825)  
at java.util.TimerThread.mainLoop(Timer.java:555)  
at java.util.TimerThread.run(Timer.java:505)  
Exception in thread "C3P0PooledConnectionPoolManager\[identityToken-\>ntek999x1778udb1lzcqhe\|12e44862\]-AdminTaskTimer" java.lang.OutOfMemoryError: unable to create new native thread  
at java.lang.Thread.start0(Native Method)  
at java.lang.Thread.start(Thread.java:717)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner.recreateThreadsAndTasks(ThreadPoolAsynchronousRunner.java:535)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner.access$400(ThreadPoolAsynchronousRunner.java:49)  
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector.run(ThreadPoolAsynchronousRunner.java:825)  
at java.util.TimerThread.mainLoop(Timer.java:555)  
at java.util.TimerThread.run(Timer.java:505)

--------------------------


Answers:
author: Igor M.
body: Hello,

Could you clarify which version of OS you are using?

There is an issue with the certain version of OS that can result in similar failures you described. The issue is outlined in [Confluence crashes and throws 'OutOfMemory: unable to create new native thread' error](https://confluence.atlassian.com/confkb/confluence-crashes-and-throws-outofmemory-unable-to-create-new-native-thread-error-423628320.html) KB.

Can you try the workaround from KB?

* Edit the $CONFLUENCE_INSTALL/bin/setenv.sh file and add the below to the top of the file:

```
ulimit -u 4096
ulimit -n 4096
```

<br />

These values may be different depending upon the operating system used, consult with your System Administrator / hosting provider about the most appropriate limits to set.

* Restart confluence.

comments: author: Kosta
body: Hi Igor,

I'm running Confluence on Ubuntu 18.04.1 LTS (Bionic Beaver).

It seems that there is no ulimit -u on user level. Only root can do.

```
root@h2798701:~# ulimit -u 4096
root@h2798701:~# su - confluence
No directory, logging in with HOME=/
$ ulimit -u 4096
-su: 1: ulimit: Illegal option -u
$ ulimit -a
time(seconds)??????? unlimited
file(blocks)???????? unlimited
data(kbytes)???????? unlimited
stack(kbytes)??????? 8192
coredump(blocks)???? 0
memory(kbytes)?????? unlimited
locked memory(kbytes) 16384
process????????????? 8192
nofiles????????????? 4096
vmemory(kbytes)????? unlimited
locks??????????????? unlimited
rtprio?????????????? 0
```

Since setenv.sh is executed as user confluence it gives an illegal option error.

So I increased "process" to 8192 in limits.conf. But it didn't change the behavior.


author: Igor M.
body: Can you also increase maxThreads in \<conf-install\>\\conf\\server.xml for the connector used, I would recommend setting it to 100 while hibernate.c3p0.max_size from \<confluence-home\>/confluence.cfg.xml should be set to at least 120 (10 counts of 25% higher rule)


author: Kosta
body: No success.

Right now I'm facing

```
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
```

errors when I try to login. A jira running on the same box is now facing also OutOfMemory unable to create new native thread errors.

A nextcloud installation seems not affected.


author: Kosta
body: [@Igor M_](/t5/user/viewprofilepage/user-id/1169056)Hi, any other hind what could block Confluence from starting and consume all resources on the machine?


author: Igor M.
body: [@Konstantinos Sianavas](/t5/user/viewprofilepage/user-id/1254741) considering that both Confluence and Jira are facing the same problem and these do not share memory or processing power in any way, this would indicate a problem with the actual server, I would go back to the problem with thread allocation, for example, this [ubuntu KB](https://askubuntu.com/questions/827458/java-lang-outofmemoryerror-unable-to-create-new-native-thread-after-upgrading)discusses the same topic


author: Craig Castle-Mead
body: Hi Konstantinos,

How much spare RAM does the server have you're trying to start Confluence on?

If this is a Linux box "free -m" should give you the answer.

CCM

comments: author: Kosta
body: > ```
> :/opt/atlassian/confluence/bin# free -m
> ????????????? total??????? used??????? free????? shared? buff/cache?? available
> Mem:????????? 12288??????? 1234?????? 10894????????? 14???????? 159??????? 6135
> Swap:???????????? 0?????????? 0?????????? 0
> ```


Tags: confluence-server

server