Confluence not working at all
Asked by Ezequiel:

Hey guys,

All of a sudden, Confluence (latest server version) stopped working and we can't bring it back to life.

Any ideas? The error we see in the logs is:

```
19-Sep-2017 20:50:36.211 SEVERE [http-nio-8090-exec-1] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=500, location=/500page.jsp]
 org.apache.jasper.JasperException: java.lang.IllegalStateException: getOutputStream() has already been called for this response
 at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:560)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:471)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.confluence.util.profiling.ConfluenceActivityFilter.doFilter(ConfluenceActivityFilter.java:39)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.confluence.web.filter.HttpRequestStatsFilter.doFilter(HttpRequestStatsFilter.java:47)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:96)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
 at com.atlassian.confluence.web.filter.ConfluenceTimingFilter.doFilter(ConfluenceTimingFilter.java:46)
 at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
 at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
 at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:716)
 at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:584)
 at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:523)
 at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:436)
 at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:301)
 at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:396)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:176)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
 at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
 at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1533)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1489)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: getOutputStream() has already been called for this response
 at org.apache.catalina.connector.Response.getWriter(Response.java:564)
 at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:212)
 at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
 at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
 at com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.getWriter(DebugFilter.java:111)
 at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:109)
 at com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.getWriter(DebugFilter.java:111)
 at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:115)
 at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:108)
 at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:176)
 at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:120)
 at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:75)
 at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:944)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
 ... 67 more
```

Thanks,

Ezequiel


Answers:
author: Leandro
body: Hi Ann, I'm Ezequiel's colleague. I'd like to thank you for the links provided, however we were unable to fix confluence following those.

We managed to get confluence working, though, here's what I did:

- Create a mysql dump of confluence DB's

```
mysqldump -h <hostname> -u <user> -p <password> --database <dbname> > <whatever-name-you-like>.sql
```

- Drop the old confluence DB. Create a new DB and grant privileges to the user, as if it was a new installation, cause that's what we're gonna do.

```
mysql -h <hostname> -u <user> -p <password>
drop database <dbname>;
create database confluence;
create user 'confluence'@'%' identified by '<password>';
grant all privileges on confluence.* to confluence;
exit
```

- Launch a new instance and install confluence on it. We installed 6.4.1. Don't terminate the old instance, you will need stuff from it!!

[Follow this link to install confluence](https://confluence.atlassian.com/doc/installing-confluence-on-linux-from-archive-file-255362363.html)

- Stop confluence and connect to the DB. Drop the new database Confluence used for the installation and source the dump from step #1.

```
mysql -h <hostname> -u <user> -p <password>
drop database confluence;
create database confluence;
source confluence-dump.sql
```

comments: author: Leandro
body: - After that, we got the following error:

```
Confluence will not start up because the build number in the home directory [7401] doesn't match the build number in the database [7400].
```

[This link will help you fix build number.](https://confluence.atlassian.com/confkb/confluence-will-not-start-up-because-the-build-number-in-the-home-directory-doesn-t-match-the-build-number-in-the-database-after-upgrade-376834096.html)

Solution #1 worked for us.

- Then we had collation and character set issues. Several of them.

[Here's how to fix collation and character set issues](https://confluence.atlassian.com/kb/how-to-fix-the-collation-and-character-set-of-a-mysql-database-744326173.html)

- Finally, we wanted all the images, thumbnails, attachments and etc we had uploaded.

Go to:

```
/var/atlassian/application-data/confluence

```

And make a backup of attachments/ and thumbnails/ directory from the old instance.

Move all this content to the new confluence instance and just uncompress it on the same path.

- Start confluence. It worked like a charm.


author: Ezequiel
body: Awesome!!


author: Shashidar
body: Hi Ann,

May i know the reason for this error in logs?

Even i'm seeing the same error in logs but my production instance is running. How can we resolve the error?

Regards,


author: AnnWorley
body: Hi Shashidar,

Which error are you seeing in which log? We discussed several errors on this thread. One was for displaying the error page, one was not being able to ping the synchrony proxy and one was concerned with database character set.

It may be better if you post a separate question so we can keep your issue distinct from this case.

Thanks,

Ann


author: Ezequiel
body: Hi Ann,

I was unable to paste the log, so here's a link to it:

<https://www.dropbox.com/s/t0k7gyvmdqf9ywv/atlassian-confluence.log?dl=0>

<br />

The weirdest thing I see is:

<br />

2017-09-21 14:38:49,623 WARN \[ListenableFutureAdapter-thread-1\] \[plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor\] pollHealthcheck Could not ping the synchrony-proxy \[<http://127.0.0.1:8090/synchrony-proxy/healthcheck>\]: {}  
java.net.SocketTimeoutException: Read timed out

comments: author: AnnWorley
body: Thanks very much for the log, it includes the whole startup, as I hoped.

I was surprised to see this at the end of the log:

```
2017-09-21 14:39:12,007 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve
```

That usually indicates that Confluence is available in the browser, although what is displayed there can turn out to be an error message.

**Synchrony Proxy**

Like you, I noticed the long stack trace related to the Synchrony health check. My expectation from that error would be issues with collaborative editing. For a useful diagram of how Synchrony works from a network perspective, please see: [Administering Collaborative Editing](https://confluence.atlassian.com/doc/administering-collaborative-editing-858772086.html#AdministeringCollaborativeEditing-Proxies) under Proxies.

**Non-recommended character set**

I was concerned to see this in the log right after start-up, as I have seen Confluence fail to start as a result of character set issues:

```
2017-09-21 14:37:42,635 WARN [localhost-startStop-1] [confluence.setup.dbcheck.MySQLChecker] checkCharacterSetAndCollation Your database uses non-recommended character set: latin1. See https://confluence.atlassian.com/x/IrYC
```

Please follow this guide to make sure the right character set is used in the Confluence database: [How to Fix the Collation and Character Set of a MySQL Database](https://confluence.atlassian.com/kb/how-to-fix-the-collation-and-character-set-of-a-mysql-database-744326173.html)


author: Ezequiel
body: Thanks Ann! Will check and let you know.


author: AnnWorley
body: Hi Ezequiel,

Please let me know more about when the instance stopped working. Did it slow down so that you restarted it and it didn't come back up? Or was the service stopped before it was restarted?

It looks like the log entry you posted is from the logs subdirectory of the Confluence installation directory. (It is often called catalina.out.) That log is for the underlying web server and will let us know if there are Tomcat configuration or other basic issues. The error you posted seems to indicate that Tomcat cannot display an error page, which is probably not the root cause of the outage.

In the [Confluence home directory](https://confluence.atlassian.com/doc/confluence-home-and-other-important-directories-590259707.html) there is another logs subdirectory, which contains the **atlassian-confluence.log**. In that log, please search for Starting Confluence and then scroll down to any errors that Confluence reports during the startup.

If the errors aren't informative enough, please post them on this thread. If I am not familiar with them I will be happy to research them and let you know my findings.

Thanks,

Ann