502 Bad Gateway after Cloud Migration
Asked by Dirk Wunderlich:

Hello Community,

We have been experiencing the following problem since last night:

We have migrated several Spaces to the cloud. This has been done last night.  
Now I wanted to look at our locally installed Confluence and got the error 502 - Bad Gateway.  
I restarted the server, but without success.

What is the best analysis path I can take here? What logs might be helpful?

I have attached in the contents of our server.xml, as well as the nginx.conf.dpkg-dist here once:

server.xml:

\<Server port="8000" shutdown="SHUTDOWN" debug="0"\>  
\<Service name="Tomcat-Standalone"\>  
\<!--  
==============================================================================================================  
DEFAULT - Direct connector with no proxy, for unproxied HTTP access to Confluence.

If using a http/https proxy, comment out this connector.  
==============================================================================================================  
--\>  
\<!--  
\<Connector port="8090" connectionTimeout="20000" redirectPort="8443"  
maxThreads="48" minSpareThreads="10"  
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"  
protocol="org.apache.coyote.http11.Http11NioProtocol"/\>  
--\>  
\<!--  
==============================================================================================================  
HTTP - Proxying Confluence via Apache or Nginx over HTTP

If you're proxying traffic to Confluence over HTTP, uncomment the connector below and comment out the others.  
Make sure you provide the right information for proxyName and proxyPort.

For more information see:  
Apache - <https://confluence.atlassian.com/x/4xQLM>  
nginx - <https://confluence.atlassian.com/x/TgSvEg>

==============================================================================================================  
--\>

\<!--  
\<Connector port="8090" connectionTimeout="20000" redirectPort="8443"  
maxThreads="48" minSpareThreads="10"  
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"  
protocol="org.apache.coyote.http11.Http11NioProtocol"  
scheme="http" proxyName="\<subdomain\>.\<domain\>.com" proxyPort="80"/\>  
--\>  

\<!--  
==============================================================================================================  
HTTPS - Direct connector with no proxy, for unproxied HTTPS access to Confluence.

For more info see <https://confluence.atlassian.com/x/s3UC>  
==============================================================================================================  
--\>

\<!--  
\<Connector port="8443" maxHttpHeaderSize="8192"  
maxThreads="150" minSpareThreads="25"  
protocol="org.apache.coyote.http11.Http11Nio2Protocol"  
enableLookups="false" disableUploadTimeout="true"  
acceptCount="100" scheme="https" secure="true"  
clientAuth="false" sslProtocol="TLSv1.2" sslEnabledProtocols="TLSv1.2" SSLEnabled="true"  
URIEncoding="UTF-8" keystorePass="\<MY_CERTIFICATE_PASSWORD\>"/\>  
--\>

\<!--  
==============================================================================================================  
HTTPS - Proxying Confluence via Apache or Nginx over HTTPS

If you're proxying traffic to Confluence over HTTPS, uncomment the connector below and comment out the others.  
Make sure you provide the right information for proxyName and proxyPort.

For more information see:  
Apache - <https://confluence.atlassian.com/x/PTT3MQ>  
nginx - <https://confluence.atlassian.com/x/cNIvMw>  
==============================================================================================================  
--\>

<br />

\<Connector port="8090" connectionTimeout="20000" redirectPort="8443"  
maxThreads="48" minSpareThreads="10"  
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"  
protocol="org.apache.coyote.http11.Http11NioProtocol"  
scheme="https" secure="true" proxyName="confluence.connect2mobile.de" proxyPort="443"/\>

\<Engine name="Standalone" defaultHost="localhost" debug="0"\>  
\<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4"\>  
\<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true"\>  
\<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties --\>  
\<Manager pathname=""/\>  
\<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/\>  
\</Context\>

\<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"  
reloadable="false" useHttpOnly="true"\>  
\<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/\>  
\</Context\>  
\</Host\>  
\</Engine\>  
\</Service\>  
\</Server\>

ngix.conf.dpkg-dist

user nginx;  
worker_processes auto;

error_log /var/log/nginx/error.log notice;  
pid /var/run/nginx.pid;

<br />

events {  
worker_connections 1024;  
}

<br />

http {  
include /etc/nginx/mime.types;  
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user \[$time_local\] "$request" '  
'$status $body_bytes_sent "$http_referer" '  
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;  
#tcp_nopush on;

keepalive_timeout 65;

#gzip on;

include /etc/nginx/conf.d/\*.conf;  
}

<br />

<br />

We have not made any changes to the system aside from migrating to the cloud. What can I check novh?

Thanks a lot!


Answers:
author: Vishwas
body: Hi [@Dirk Wunderlich](/t5/user/viewprofilepage/user-id/2046872)

server.xml looks fine.

You can start looking at confluence log files it will be like this in confluence home directory.

`path: <confluence-home>/logs/atlassian-confluence.log.`

If you can paste the ERROR from log file we can see what is the exact issue.

Usually 502 error bad gateway ,that is your reverse proxy server is running but confluence has some issues starting.

Regards,

Vishwas

comments: author: Dirk Wunderlich
body: Hi [@Vishwas](/t5/user/viewprofilepage/user-id/1231381)

thanks for your response. The log is currently uhm... a bit strange... Here the last lines. Some entries this night, after this a big nothing and for now two lines:

2021-12-18 04:07:56,848 WARN \[Caesium-1-1\] \[impl.schedule.caesium.JobRunnerWrapper\] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse\[runOutcome=ABORTED,message='LaaS performance logging is turned off'\]  
2021-12-18 04:12:56,849 WARN \[Caesium-1-4\] \[impl.schedule.caesium.JobRunnerWrapper\] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse\[runOutcome=ABORTED,message='LaaS performance logging is turned off'\]  
2021-12-18 08:34:38,582 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 7.2.0 \[build 8401 based on commit hash 912a577971fe711d4a876d9130277d03b009a0bf\] - synchrony version 3.1.0-master-0f086db5  
2021-12-18 08:34:46,526 INFO \[Catalina-utility-1\] \[springframework.web.context.ContextLoader\] initWebApplicationContext Root WebApplicationContext: initialization started  
2021-12-18 08:34:55,415 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] \<init\> Loading EhCache cache manager  
2021-12-18 13:14:47,445 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 7.2.0 \[build 8401 based on commit hash 912a577971fe711d4a876d9130277d03b009a0bf\] - synchrony version 3.1.0-master-0f086db5  
2021-12-18 13:14:51,093 INFO \[Catalina-utility-1\] \[springframework.web.context.ContextLoader\] initWebApplicationContext Root WebApplicationContext: initialization started  
2021-12-18 13:15:16,981 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 7.2.0 \[build 8401 based on commit hash 912a577971fe711d4a876d9130277d03b009a0bf\] - synchrony version 3.1.0-master-0f086db5  
2021-12-18 13:15:20,745 INFO \[Catalina-utility-1\] \[springframework.web.context.ContextLoader\] initWebApplicationContext Root WebApplicationContext: initialization started  
2021-12-18 13:15:25,730 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] \<init\> Loading EhCache cache manager  
2021-12-18 13:15:42,473 INFO \[Catalina-utility-1\] \[springframework.web.context.ContextLoader\] initWebApplicationContext Root WebApplicationContext initialized in 21727 ms  
2021-12-18 13:15:43,904 INFO \[Catalina-utility-1\] \[atlassian.plugin.manager.DefaultPluginManager\] lambda$earlyStartup$3 Plugin system earlyStartup begun  
2021-12-18 13:15:45,155 WARN \[Catalina-utility-1\] \[org.twdata.pkgscanner.ExportPackageListBuilder\] logDuplicateWarning Package Scanner found duplicates for package 'com.atlassian.integration.jira.applinks' with different versions. Files: jira-integration-spi-6.2.0.jar and confluence-7.2.0.jar  
'/opt/atlassian/confluence/confluence/WEB-INF/lib/jira-integration-spi-6.2.0.jar'  
'/opt/atlassian/confluence/confluence/WEB-INF/lib/confluence-7.2.0.jar'  
2021-12-18 13:15:48,358 WARN \[FelixStartLevel\] \[extender.internal.support.ExtenderConfiguration\] addDefaultDependencyFactories Gemini Blueprint extensions bundle not present, annotation processing disabled.

The last ERROR we recieved this night at 03:15:

2021-12-18 03:15:50,379 ERROR \[http-nio-8090-exec-11\] \[confluence.plugins.synchrony.SynchronyContextProvider\] getContextMap  
-- url: /pages/createpage-entervariables.action \| traceId: 486a6ea2d1d82995 \| userName: anonymous \| action: createpage-entervariables  
2021-12-18 03:15:51,547 ERROR \[http-nio-8090-exec-8\] \[confluence.plugins.synchrony.SynchronyContextProvider\] getContextMap  
-- url: /pages/createpage-entervariables.action \| traceId: ec6fb2917277e678 \| userName: anonymous \| action: createpage-entervariable

Thank you and best regards

Dirk


author: Vishwas
body: Hi [@Dirk Wunderlich](/t5/user/viewprofilepage/user-id/2046872)

This looks odd, the recent logs shows INFO and WARN levels. Seems no ERROR recently.

Can you check the status of the nginx services and send the results of *ps -ef \| grep confluence*

Try restarting nginx and confluence services.

Regards,

Vishwas


author: Dirk Wunderlich
body: Hi [@Vishwas](/t5/user/viewprofilepage/user-id/1231381) ,

yes, its really strange :(

nginx is running as far as I can see it.

![nginx_service.jpg](https://community.atlassian.com/t5/image/serverpage/image-id/174626i8A627821C554F48F/image-size/large?v=v2&px=999 "nginx_service.jpg")  
I restarted it anyway, without success. JIRA and BitBucket are still running on the same machine, they work without errors.

The output from the ps -ef is the follwing:  
![grep.jpg](https://community.atlassian.com/t5/image/serverpage/image-id/174627iEB258DC5DBCB1B6B/image-size/large?v=v2&px=999 "grep.jpg")

I also restarted the confluence-service, but without success.

Thank you and best regards

Dirk


author: Vishwas
body: Hi [@Dirk Wunderlich](/t5/user/viewprofilepage/user-id/2046872)

Really not able to figure out what is the issue :(

Can you please contact Atlassian Support Team they can assist better by looking at full logs on this here <https://support.atlassian.com/contact/#/>

Also provide them with support zip. Steps to create support zip :

Go to **Settings(Cog Wheel) \> General Configuration \> Troubleshooting and support tools**(under Administration)

Regards,

Vishwas


author: Dirk Wunderlich
body: Hi [@Vishwas](/t5/user/viewprofilepage/user-id/1231381)

thank you! Yes, I wil raise a ticket.   
But unfortunately I am not able to raise a support zip, because I can't reach my confluence :D It is a vicious circle...

Thank you and best regards!

Dirk


author: Vishwas
body: Hi [@Dirk Wunderlich](/t5/user/viewprofilepage/user-id/2046872)

I forgot this my bad. Apologies.

Atlassian will ask for specific log files from few folders you can provide them once they reply to the ticket.

Regards,

Vishwas


author: Dirk Wunderlich
body: Hi [@Vishwas](/t5/user/viewprofilepage/user-id/1231381) ,

no problem, really! Thanks for your assistance in this case!

Best regards

Dirk


author: Dirk Wunderlich
body: Hi [@Vishwas](/t5/user/viewprofilepage/user-id/1231381)

i got it. It seems that there was some maleware on our server which claimed much off ressources.

I cleaned it up and now Confluence is up again!

Thank you again for your support. With this I was able to walk in the correct direction.

Best regards

Dirk


author: Vishwas
body: Hi [@Dirk Wunderlich](/t5/user/viewprofilepage/user-id/2046872)

Glad to hear the issue got fixed !!

Thanks,

Vishwas


Tags: confluence-server

server