High CPU Usage - isolating process
Asked by IT KnowledgeSystemsManagement:

We recently experienced an hour long period of extremely high (90%+) CPU usage on our 16 core server which abide after about an hour. During this time no users reported any issues accessing Confluence and monitoring information from Wily indicated that response times were normal. For all intensive purposes the system was functional and within normal tolerances. However we would like to isolate what caused the CPU spike and were wondering if there is anyway to isolate which process would have caused this.

Regards,

-Christopher


Answers:
author: Simon Leclercq
body: i have the same problem as i well


author: Steve Petroskey
body: Hi,

We are suddenly seeing the same issue. Running top, the "khugepageds" process owned by confluence is pegged, reporting 198-200% CPU, load average of 2.3, and User CPU over 30%.

This is running on an AWS EC2, which I doubled, only to see it run fine for less than an hour and then max out again. (Which kicks in AWS throttling, so it would probably consume all the CPU if it could.)

Before I doubled the CPU, the application wouldn't respond. After that, it works with a slight delay, but is still maxed out. Throwing more CPU at it will soon become unacceptable.

comments: author: David White
body: [@Steve Petroskey](/t5/user/viewprofilepage/user-id/902462), in your situation "khugepageds" is almost certainly a rogue cryptocurrency miner, which someone has installed on your system via the [WebDAV and Widget vulnerability](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html?_ga=2.16158710.1140342368.1555169218-1974102225.1555169218)

Kill the khugepageds processes, delete the rogue crontab entry for "confluence" that is running curl/wget processes, delete /tmp/kerberods (which has probably opened reverse SSH tunnels on your system), and reboot. Then patch your system.


author: Patrick du Boucher
body: Hello,

I don\`t have an answer, but we have exactly the same problem.

Twice in the last three weeks, the CPU went from 10% to about 30% for a couple of hours, and then spiked to 100% and never came down.

During this time, There were a lot of connections to the DB that were all idle, and usage in the application seemed unaffacted. Even collarborative editing was behaving correctly.

rebooting brought the CPU back to normal, but I don\`t have a resolution, or explanation.

thanks

comments: author: DKB Service GmbH
body: hi, we are facing absolutly the same issue since last week. do you have a resolution


author: Fusion Helpdesk
body: Hi,

Confluence Cpu: 13.7% and back to normal after rebooted server.

Thanks


author: Fusion Helpdesk
body: Hi Christopher,

We have submitted atlassian support ticket for this (<https://getsupport.atlassian.com/servicedesk/customer/portal/14/CSP-202831>) and monitoring very closely.

Thanks


author: Joe Clark
body: Hi Christopher,

Do you know if the spike is coming from Confluence, or some other process on your system?

For Confluence, the best thing to do is, when the spike starts happening begin taking a bunch of thread dumps using the [built-in admin tool](http://confluence.atlassian.com/display/DOC/Generating+a+Thread+Dump) spaced 5-10 seconds apart and then submit them to <https://support.atlassian.com> for analysis.

If the spike is happening outside of Confluence, you'll need to resort to other ways that are dependent on your operating system. For Windows, you could use the built in performance monitoring tools to plot CPU usage by process over time and then see which process is spiking.

For \*nix systems, you can use the *top* command.