JIRA, Confluence, Crowd, Stash(bitbucker) don't boot up.
Asked by Krzysztof Macewicz:

I need some help with a very annoying problem.

My company uses the Atlassian stack: JIRA, Bitbucker, Confluence and Crowd. We got all of them in one remote server. To log into the applications we use Nginx.

So the problem is:

Often (one time per two days) all the atlassian applications stop working. It's really strange, when I try to log into JIRA it says 'Server internal error', on other applications (BB, Crowd and Conf) I got an error that Nginx didn't find any applications under proxy. I'm really confused about this issue. For now I solve it by restarting all applications (there are scripts inside the application to restart it (stop and start)), then I reboot the PC and restart Nginx. Sometimes the applications are up after first reboot, sometimes after third or even fourth reboot. It's very annoying. Nginx is not a problem, because other applications like nexus are working. On my remote server I use ubuntu 16.04.

My questions are:

- Is there any procedure to check health of the applications?  
- Why do these applications stop working and crash so often?  
- Is there any good practice to manage this problem?

Cheers


Answers:
author: Marcin Kempa
body: Hi [@Krzysztof Macewicz](/t5/user/viewprofilepage/user-id/1166579),

Could you post some of your logs when you experience downtime? That would help us to understand the problem.

Cheers,

Marcin Kempa

comments: author: Krzysztof Macewicz
body: Hi Marcin,

Yes, of course. Actually today morning it stopped working so the log is fresh. Yesterday it was working till 7 pm. It was the time when last person left office. My colleges told me that JIRA is not working since 9 AM.

I sent you screen when i try to connect jira.![Screenshot from 2017-11-28 10-39-24.png](https://community.atlassian.com/t5/image/serverpage/image-id/10089i21EF5EE012FA4E7C/image-size/large?v=v2&px=999 "Screenshot from 2017-11-28 10-39-24.png")

Can you please give me e-mail to you? Log has 126,6 kB.

Cheers,

Krzysztof


author: Marcin Kempa
body: Sure, my email is: mkempa@atlassian.com


author: Krzysztof Macewicz
body: Hi Christian,

Yes, of course.

disk usage (df -h):

Filesystem Size Used Avail Use% Mounted on  
/dev/mapper/vg_atlassian-lv_root 64G 59G 2,2G 97% /  
tmpfs 11G 0 11G 0% /dev/shm  
/dev/sda1 485M 34M 426M 8% /boot  
/dev/mapper/vg_atlassian-lv_home 45G 9,9G 33G 24% /home

Memory (watch -n 5 free -m)

total used free shared buffers cached  
Mem: 22007 12899 9107 0 124 5939  
-/+ buffers/cache: 6836 15171  
Swap: 3967 0 3967

The system looks like is CentOS, sorry for mistake.

cat /etc/\*release  
CentOS release 6.5 (Final)  
CentOS release 6.5 (Final)  
CentOS release 6.5 (Final)

comments: author: Lars Olav Velle
body: Is your nexus service is running as a different user?

My guess is that the problem is too many open files.

<https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/>

-Lars


author: Krzysztof Macewicz
body: Thanks Lars for suggestion.

I checked max files that can be open, result:

cat /proc/sys/fs/file-max  
2237374

Should i increase it?

My Nexus is running on diffrent virtual machine.


author: Lars Olav Velle
body: The /proc/sys/fs/file-max would be the maximum allowed for the whole server. Leave that as is.

Here is output from a server with increased maximum open files for the user otherUser:

\[root@server \~\]# ulimit -Hn  
4096  
\[root@server \~\]# ulimit -Sn  
1024

\[root@server \~\]# su otherUser

\[otherUser@server\]$ ulimit -Sn  
131072  
\[otherUser@server\]$ ulimit -Hn  
131072

These settings you\`ll find in:

/etc/security/limits.conf

otherUser hard nofile 131072  
otherUser soft nproc 131072  
otherUser hard nproc 131072  
otherUser soft nofile 131072  
# End of file


author: Krzysztof Macewicz
body: Thanks for reply

I run as root.

\[root@Atlassian \~\]# ulimit -Hn  
4096  
\[root@Atlassian \~\]# ulimit -Sn  
1024

Is it correct?

Open: `/etc/security/limits.conf`

`add: root soft nofile 4096`  
` root hard nofile 10240`

`Cheers,`

`Krzysztof`


author: Lars Olav Velle
body: Actually I am not entirely sure as I have never run any applications as root.

If you want to determine if this is the actual problem you might want to test with unlimited first.

The following will set unlimited, but will be back to standard after a reboot.

(su - username)

ulimit


author: Krzysztof Macewicz
body: You're right. I haven't configure this machine.

There is a user jira and crowd.

Both got

ulimit -Hn  
4096

ulimit -Sn  
1024  

To make this users unlimit, I should:

-log into root user

-open `/etc/security/limits.conf`

-add: `jira soft nofile unlimited`

` jira hard nofile unlimited`

`same for crowd user.`

`Am I correct?`

`Thanks a lot`

`Krzysztof`


author: Christian Glockner
body: Hi Krzsysztof,

Can you give more details about your system, such as how much RAM you have on the machine?

Cheers,

Christian

Premier Support Engineer

Atlassian

comments: author: Krzysztof Macewicz
body: Hi Christian,

Yes, of course.

disk usage (df -h):

Filesystem Size Used Avail Use% Mounted on  
/dev/mapper/vg_atlassian-lv_root 64G 59G 2,2G 97% /  
tmpfs 11G 0 11G 0% /dev/shm  
/dev/sda1 485M 34M 426M 8% /boot  
/dev/mapper/vg_atlassian-lv_home 45G 9,9G 33G 24% /home

Memory (watch -n 5 free -m)

total used free shared buffers cached  
Mem: 22007 12899 9107 0 124 5939  
-/+ buffers/cache: 6836 15171  
Swap: 3967 0 3967

The system looks like is CentOS, sorry for mistake.

cat /etc/\*release  
CentOS release 6.5 (Final)  
CentOS release 6.5 (Final)  
CentOS release 6.5 (Final)


Tags: bitbucket

confluence

jira

jira-service-desk