confluence 6.15 stopped working
Asked by pierrexor:

Hi,

for whatever reason our confluence stopped working as in gateway timeout in nginx. Curl also doesnt get anything back, telnet can connect but nothing else. Now that happens but I am stuck because I can't find the reason.

System is clear, Java runs and starts fine. It's listening:

tcp6 0 0 :::8090 :::\* LISTEN 2271/java

Log

10-Jul-2019 08:46:02.298 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server\] failed to set property \[debug\] to \[0\]  
10-Jul-2019 08:46:02.358 WARNING \[main\] org.apache.catalina.startup.SetAllPropertiesRule.begin \[SetAllPropertiesRule\]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.  
10-Jul-2019 08:46:02.374 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine\] failed to set property \[debug\] to \[0\]  
10-Jul-2019 08:46:02.380 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host\] failed to set property \[debug\] to \[0\]  
10-Jul-2019 08:46:02.408 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context\] failed to set property \[debug\] to \[0\]  
10-Jul-2019 08:46:02.431 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context\] failed to set property \[debug\] to \[0\]  
10-Jul-2019 08:46:02.778 INFO \[main\] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler \["http-nio-8090"\]  
10-Jul-2019 08:46:02.814 INFO \[main\] org.apache.catalina.startup.Catalina.load Server initialization in \[607\] milliseconds  
10-Jul-2019 08:46:02.823 INFO \[main\] org.apache.catalina.core.StandardService.startInternal Starting service \[Tomcat-Standalone\]  
10-Jul-2019 08:46:02.824 INFO \[main\] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: \[Apache Tomcat/9.0.19\]  
10-Jul-2019 08:46:04.825 INFO \[Catalina-utility-2\] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: \[com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@245979d5\]  
10-Jul-2019 08:46:04.983 INFO \[Catalina-utility-2\] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.  
2019-07-10 08:46:05,766 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 6.15.6 \[build 8100 based on commit hash e533a30abc9e1cf90ba3519e12647de186f0ee76\] - synchrony version 2.1.0-release-confluence_6.15-32f7299a  
10-Jul-2019 08:47:29.667 WARNING \[Catalina-utility-2\] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using \[SHA1PRNG\] took \[84,678\] milliseconds.  
10-Jul-2019 08:47:29.850 INFO \[Catalina-utility-2\] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'  
10-Jul-2019 08:47:29.850 INFO \[Catalina-utility-2\] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started  
10-Jul-2019 08:47:29.856 INFO \[Catalina-utility-2\] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date \[Wed Jul 10 08:47:29 UTC 2019\]; root of context hierarchy  
10-Jul-2019 08:47:29.904 INFO \[Catalina-utility-2\] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: \[class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig\]  
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".  
SLF4J: Defaulting to no-operation (NOP) logger implementation  
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.  
10-Jul-2019 08:47:30.340 INFO \[Catalina-utility-2\] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'  
10-Jul-2019 08:47:30.381 INFO \[Catalina-utility-2\] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path \[/v1/bayeux-sync1\] onto handler of type \[class org.springframework.web.socket.server.support.WebSocketHttpRequestHandler\]  
10-Jul-2019 08:47:30.732 INFO \[Catalina-utility-2\] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{\[/healthcheck\]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()  
10-Jul-2019 08:47:30.734 INFO \[Catalina-utility-2\] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{\[/reload\],methods=\[PUT\]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)  
10-Jul-2019 08:47:30.761 INFO \[Catalina-utility-2\] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path \[/\*\*\] onto handler of type \[class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler\]  
10-Jul-2019 08:47:30.790 INFO \[Catalina-utility-2\] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for @ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date \[Wed Jul 10 08:47:29 UTC 2019\]; root of context hierarchy  
10-Jul-2019 08:47:30.893 INFO \[Catalina-utility-2\] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647  
10-Jul-2019 08:47:30.915 INFO \[Catalina-utility-2\] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1065 ms

We're using nginx as proxy, nginx config is identical to the suggested one on confluence. I also checked all related config files.

server {  
listen \*:443;  
server_name wiki.xxxxx;  
ssl on;  
ssl_certificate /etc/ssl/xxxxt;  
ssl_certificate_key /etc/ssl/xxxxx;  

location / {  
client_max_body_size 100m;  
proxy_set_header X-Forwarded-Host $host;  
proxy_set_header X-Forwarded-Server $host;  
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
proxy_pass http://localhost:8090/;  
}  
location /synchrony {  
proxy_set_header X-Forwarded-Host $host;  
proxy_set_header X-Forwarded-Server $host;  
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
proxy_pass http://localhost:8091/synchrony;  
proxy_http_version 1.1;  
proxy_set_header Upgrade $http_upgrade;  
proxy_set_header Connection "Upgrade";  
}  
}


Answers:
author: Gonchik Tsymzhitov
body: Hi!

can you open site directly through 8090 port ?

Cheers,

Gonchik Tsymzhitov

comments: author: pierrexor
body: no, that was the comment about curl etc.

times out


author: Gonchik Tsymzhitov
body: I have asked to understand is it related to the nginx side or tomcat side?

Because **proxy_read_timeout** is 60s by default, and telnet works well if you tried.

and logs don't show any errors.

If I were you, i will troubleshoot first without nginx and then with nginx


author: pierrexor
body: edit: I figured out whats wrong via strace

<br />

Not even proper error reporting does this software has...seriously what are the guys at Atlassian doing...


Tags: confluence-server

server