502 Bad Gateway
Asked by arcadiaengineering:

This morning, due to no action on our part, our confluence installation on aws shows a 502: Bad Gateway error. Stopping and restarting the EC2 instances where confluence is installed solves the problem....temporarily. Shortly afterwards (in a couple of hours), the site goes down with the same error again.

Everything was working fine up until yesterday, we've made no changes to the AWS setup or installation files at all. How can we fix this?


Answers:
author: arcadiaengineering
body: Formal answer: we were attacked from the vulnerability detailed here: <https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337>

Following the steps in that thread fixed it for us (kill hkugepads process, clear cron jobs).


author: Diego
body: Hello there!

Could you clarify some points for us?
> 1. Which version of Confluence are you running?
> 2. Is CPU usage unusually high in your server?

Looking forward to your reply!

comments: author: arcadiaengineering
body: Hi, thanks for offering to help. Answers below.

1. How do I find my confluence version? I cannot access confluence from the url, so I can't just go to the admin panel and find out (bad gateway error). But I can access it via the command line. How do I find out/what command/which file?

2. CPU usage according to AWS is at 100%. That's pretty high, right? Unfortunately I don't have statistics of CPU usage before this morning, because when I stopped and restarted my AWS instances to see if I could solve the problem, they were somehow automagically terminated and re spun/re-built. So the ones I currently have running are somehow "new" ones. Stopping/starting the AWS instances solved the bad gateway problem for an hour or two, then it came back.


author: arcadiaengineering
body: Our version is 6.12.2 .

I was able to get confluence back up by following instructions in this thread:

<https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337>


author: Diego
body: Hey there!

Great to know that you solved the issue! I was a little late to reply here but thankfully everything is ok.

Thanks for sharing your Connie version. With this information I can recommend you to upgrade your instance to at least 6.12.4. This is due to these two CVEs:

**[Confluence Security Advisory - 2019-03-20 \| March 2019 Confluence Server Advisory - WebDAV and Widget Connector vulnerabilities](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html)**

**[Confluence Security Advisory - 2019-04-17 \| Confluence - Path traversal vulnerability - CVE-2019-3398](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-04-10-968660855.html)**

Also, try running another malware check on your instance just to be sure that everything is in order.

Further! Before the upgrade or any other change you may apply to your instance, backup these:

* \<confluence-home\> folder
* \<confluence-install\> folder
* Database

More information:

**[Confluence Home and other important directories](https://confluence.atlassian.com/doc/confluence-home-and-other-important-directories-590259707.html)**

Let us know your thoughts!


author: arcadiaengineering
body: I just ran top on the confluence server. It claims command khugepageds is taking 399% of the CPU. Is this helpful info? Not sure what to do next...

comments: author: arcadiaengineering
body: Looks like we may be suffering from this same attack: <https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337>

Following the steps in that thread, will post if still need more assistance. Thx.


author: Gary Pasquale
body: Hi [@arcadiaengineering](/t5/user/viewprofilepage/user-id/3050388) ,

A 502 error isn't going to tell you much with further investigation.

I'd start by following this guide:

<https://blog.hubspot.com/marketing/502-bad-gateway>

That will be a good starting point.

Regards,

Gary


Tags: atl-confsec

aws

confluence-server

server