strange behavior
Asked by kreativkonzept:

we had to restart our confluence server as there was a process "dblaunchs" with 3 threads which consumed all CPU cycles and stopping did not work.

After restart Confluence came up ok, I could log in, but after a few minutes a process started which in "top" is blank (no command), using "htop" there are three threads of the confluence user who execute the command "/boot/vmlinuz" and again, these three consume all CPU cycles and make the system non-reachable.

Any ideas?


Answers:
author: Daniel Eads
body: Based on your symptoms, it sounds like your instance was affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th ([see Confluence Security Advisory - 2019-03-20](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html)). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.

I'd recommend tackling things in this order:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

**Malicious processes**

The **top** command will help you find processes (probably running under the confluence user account) that are consuming a large amount of CPU. If Confluence is currently stopped, you can probably plan on killing any processes running as the confluence user. note the process ID (pid) from the **top** output and then kill the process using **kill -9** followed by the pid. Example:

```
sudo kill -9 12395
```

**Clean up your crontab**

Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to [check the crontab file](https://help.ubuntu.com/community/CronHowto) and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.

Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.

```
sudo vim /var/spool/cron/crontabs/confluence
```

Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to **not exist** under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.

**Upgrade Confluence**

Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

* 6.13.4 (6.13 is an [Enterprise release](https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html))
* 6.14.3
* 6.15.2

**Use a malware scanner**

Finally, you need to clean up any remaining traces of malware on your system. The [LSD malware cleanup tool](https://git.laucyun.com/security/lsd_malware_clean_tool/blob/master/README.md) will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections [are the scanners linked here](https://www.tecmint.com/scan-linux-for-malware-and-rootkits/). Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.

Please let me know if you have more questions!  
Daniel \| Atlassian Support

comments: author: kreativkonzept
body: Hi David,

that was the way to go, we got rid of it.

If only updating would preserve the SSL config, that part can be a pain in the ass, especially if you're in a hurry...

Best regards,

Lucian


author: Whit Blauvelt
body: The LSD toolkit is best used as suggestion. It is just simple scripts, that do things like remove root's cron file -- which is neither necessary nor desirable unless the attacker has managed privilege escalation, which is not reportedly a normal aspect with this attack kit. So look at what the scripts do, and adapt them. Running them blindly is questionable advice.


author: TOSM Enterprise
body: [@kreativkonzept](/t5/user/viewprofilepage/user-id/1671942) are you still seeing this strange behavior. We are seeing this behavior as well.

However when we stop confluence and kill the process we cannot restart confluence.


author: Lucinda Stroud
body: We are also encountering this. I've used the kill -9 command on the dblaunchs, but was unable to find any crontabs. Within just a couple of minutes, dblaunchs reasserts itself.

Using

```
crontab -u confluence -l
```

brings up no results of crontabs for the user confluence.

I'm working off a fresh upgrade to 6.14.3, which should have the patch resolved.

I've run all of the scripts on the [LSD Malware Tool](https://git.laucyun.com/security/lsd_malware_clean_tool/tree/master) as well just in case, but no success.

comments: author: Lucinda Stroud
body: Update: I was able to solve this by

* Upgrading all the way to 6.15.2
* Killing the dslaunchs process
* Killing the sh process
  * Both were being managed by a "confluence" user even though no cronjobs came up for that user


Tags: atl-confsec

confluence-server

server