Confluence process killed at startup
Asked by tommynator:

We are running self-hosted version of Confluence 6.2 on Ubuntu 16.04.

Since today the service won't start anymore but is instantly killed after confluence-start is called. The catalina.out log shows no errors, just some warnings.

```
---------------------------------------------------------------------------
Using Java: /home/confluence/atlassian/confluence/jre//bin/java
2019-04-15 12:15:52,840 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /home/confluence/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /home/confluence/atlassian/confluence
Using CATALINA_HOME: /home/confluence/atlassian/confluence
Using CATALINA_TMPDIR: /home/confluence/atlassian/confluence/temp
Using JRE_HOME: /home/confluence/atlassian/confluence/jre/
Using CLASSPATH: /home/confluence/atlassian/confluence/bin/bootstrap.jar:/home/confluence/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_PID: /home/confluence/atlassian/confluence/work/catalina.pid
15-Apr-2019 12:15:53.403 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 12:15:53.594 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 12:15:53.618 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 12:15:53.638 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 12:15:53.689 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 12:15:53.720 WARNING [main] org.apache.catalina.core.StandardContext.setPath A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [${confluence.context.path}/synchrony-proxy] does not meet these criteria and has been changed to [/${confluence.context.path}/synchrony-proxy]
15-Apr-2019 12:15:53.721 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 12:15:53.725 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslProtocols' to 'TLSv1,TLSv1.1,TLSv1.2' did not find a matching property.
15-Apr-2019 12:15:54.304 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
15-Apr-2019 12:15:54.332 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
15-Apr-2019 12:15:54.336 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8443"]
15-Apr-2019 12:15:54.959 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
15-Apr-2019 12:15:54.959 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1671 ms
15-Apr-2019 12:15:54.977 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
15-Apr-2019 12:15:54.977 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.41
Killed
```

All other confluence/logs are either empty or don't show any error, stacktrace or something else useful.

Any ideas on how to debug this issue?

Thanks,

Thomas


Answers:
author: Daniel Eads
body: Hi Thomas,

Based on your version and symptoms, it sounds like your instance was affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th ([see Confluence Security Advisory - 2019-03-20](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html)). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.

I'd recommend tackling things in this order:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

**Malicious processes**

The **top** command will help you find processes (probably running under the confluence user account) that are consuming a large amount of CPU. If Confluence is currently stopped, you can probably plan on killing any processes running as the confluence user. note the process ID (pid) from the **top** output and then kill the process using **kill -9** followed by the pid. Example:

```
sudo kill -9 12395
```

**Clean up your crontab**

Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to [check the crontab file](https://help.ubuntu.com/community/CronHowto) and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.

Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.

```
sudo vim /var/spool/cron/crontabs/confluence
```

Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to **not exist** under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.

**Upgrade Confluence**

Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

* 6.13.4 (6.13 is an [Enterprise release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html))
* 6.14.3
* 6.15.2

**Use a malware scanner**

Finally, you need to clean up any remaining traces of malware on your system. The [LSD malware cleanup tool](https://git.laucyun.com/security/lsd_malware_clean_tool/blob/master/README.md) will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections [are the scanners linked here](https://www.tecmint.com/scan-linux-for-malware-and-rootkits/). Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.

Please let me know if you have more questions!  
Daniel \| Atlassian Support

comments: author: tommynator
body: Thanks for the detailed description. It was exactly the issue you mentioned. After removing any traces of the malware and upgrading confluence to 6.15.2 the problem was resolved and everything went back to normal.

Cheers,

Thomas


author: Alex Ravary
body: I just fixed my issue thanks for this post, after a long number of hours of searching and googling. Thank you for this detailed post to such a simple solution.

Check your **crontabs** people!

Cheers,

Alex


Tags: atl-confsec

confluence-server

server