Issue upgrading confluence from CentOS 7 6.0.4 to Ubuntu 16.04 6.5.0 with Galera cluster.
Asked by Robert Jackson:

Hello all,

I am having some issues with upgrading my confluence server as previously stated. I am changing distributions as well although I don't think that this would cause an issue. One notable difference that I have this go around is that I am using a highly available MySQL 5.6 3 node cluster via Galera.

![Screen-Shot-2018-04-11-at-12.47.12-AM.png](https://community.atlassian.com/t5/image/serverpage/image-id/19324i6563C10CD531C70D/image-size/large?v=v2&px=999 "Screen-Shot-2018-04-11-at-12.47.12-AM.png")

Hera is the corresponding log events for this:

```
2018-04-11 05:38:17,805 ERROR [Long running task: Importing data] [engine.jdbc.spi.SqlExceptionHelper] logExceptions No operations allowed after connection closed.
 -- url: /setup/longrunningtaskxml.action | traceId: 6a5a74143e20b185 | userName: anonymous | referer: http://grimoire.scriptmyjob.com/setup/setup-restore-progress.action?taskId=913b9549-4451-4f67-bc54-7f259ff156c9 | action: longrunningtaskxml
2018-04-11 05:38:17,829 WARN [Long running task: Importing data] [confluence.upgrade.impl.DefaultUpgradeManager] setDatabaseBuildNumber Unable to set build number '7502' in the database
 -- url: /setup/longrunningtaskxml.action | traceId: 6a5a74143e20b185 | userName: anonymous | referer: http://grimoire.scriptmyjob.com/setup/setup-restore-progress.action?taskId=913b9549-4451-4f67-bc54-7f259ff156c9 | action: longrunningtaskxml
org.springframework.dao.DataAccessResourceFailureException: could not prepare statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not prepare statement
 at org.springframework.orm.hibernate5.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:220)
 at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:366)
 at org.springframework.orm.hibernate5.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:326)
 at org.springframework.orm.hibernate5.HibernateTemplate.findByNamedQueryAndNamedParam(HibernateTemplate.java:989)
 at org.springframework.orm.hibernate5.HibernateTemplate.findByNamedQueryAndNamedParam(HibernateTemplate.java:978)
 at com.atlassian.confluence.core.persistence.hibernate.HibernateVersionHistoryDao.getVersionHistory(HibernateVersionHistoryDao.java:44)
 at com.atlassian.confluence.core.persistence.hibernate.HibernateVersionHistoryDao.addBuildToHistory(HibernateVersionHistoryDao.java:116)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:16)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
 at com.sun.proxy.$Proxy161.addBuildToHistory(Unknown Source)
 at com.atlassian.confluence.upgrade.impl.DefaultUpgradeManager.setDatabaseBuildNumber(DefaultUpgradeManager.java:200)
 at com.atlassian.confluence.upgrade.impl.DefaultUpgradeManager.setDatabaseBuildNumber(DefaultUpgradeManager.java:187)
 at com.atlassian.confluence.importexport.xmlimport.BackupImporter.doUpgrades(BackupImporter.java:354)
 at com.atlassian.confluence.importexport.xmlimport.BackupImporter.doImportInternal(BackupImporter.java:229)
 at com.atlassian.confluence.importexport.Importer.doImport(Importer.java:73)
 at com.atlassian.confluence.importexport.DefaultImportExportManager.performImportInternal(DefaultImportExportManager.java:131)
 at com.atlassian.confluence.importexport.DefaultImportExportManager.doPerformImport(DefaultImportExportManager.java:119)
 at com.atlassian.confluence.importexport.DefaultImportExportManager.performImport(DefaultImportExportManager.java:114)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
 at com.sun.proxy.$Proxy243.performImport(Unknown Source)
 at com.atlassian.confluence.importexport.actions.ImportLongRunningTask.runInternal(ImportLongRunningTask.java:74)
 at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:26)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at com.atlassian.confluence.tenant.TenantGate$3.call(TenantGate.java:145)
 at com.atlassian.confluence.tenant.OpenTenantGateLongRunningTask.run(OpenTenantGateLongRunningTask.java:24)
 at com.atlassian.confluence.util.longrunning.ManagedTask.run(ManagedTask.java:52)
 at com.atlassian.confluence.impl.util.concurrent.ConfluenceExecutors$ThreadLocalContextTaskWrapper.lambda$wrap$1(ConfluenceExecutors.java:90)
 at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:50)
 at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
 at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
 at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:49)
 at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$withRequestContext$2(VCacheRequestContextOperations.java:66)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
 at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
 at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
 at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
 at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)
 at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:148)
 at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1940)
 at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1909)
 at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887)
 at org.hibernate.loader.Loader.doQuery(Loader.java:932)
 at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
 at org.hibernate.loader.Loader.doList(Loader.java:2615)
 at org.hibernate.loader.Loader.doList(Loader.java:2598)
 at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430)
 at org.hibernate.loader.Loader.list(Loader.java:2425)
 at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:502)
 at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:370)
 at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216)
 at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1481)
 at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1441)
 at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1410)
 at org.springframework.orm.hibernate5.HibernateTemplate$33.doInHibernate(HibernateTemplate.java:1001)
 at org.springframework.orm.hibernate5.HibernateTemplate$33.doInHibernate(HibernateTemplate.java:989)
 at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:359)
 ... 58 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
 at com.mysql.jdbc.Util.getInstance(Util.java:408)
 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
 at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1184)
 at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1179)
 at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4071)
 at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4040)
 at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:387)
 at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:146)
 at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:172)
 ... 77 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 145,197 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago.
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
 at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
 at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
 at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
 at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
 at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
 at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2480)
 at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2438)
 at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1381)
 at com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:327)
 at com.atlassian.confluence.core.persistence.hibernate.HibernateVersionHistoryDao.getLatestBuildNumber(HibernateVersionHistoryDao.java:30)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:16)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
 at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
 at com.sun.proxy.$Proxy161.getLatestBuildNumber(Unknown Source)
 at com.atlassian.confluence.upgrade.impl.DefaultUpgradeManager.setDatabaseBuildNumber(DefaultUpgradeManager.java:197)
 ... 37 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
 at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3011)
 at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3469)
 ... 62 more
```

Here is my Infrastructure as code for standing up the Galera Cluster:  
<https://github.com/ScriptMyJob/Galera_Cluster>

And here is my IaC for the Confluence App server:  
<https://github.com/ScriptMyJob/Grimoire>

Any help would be very much appreciated.

Repsectfully,  
Robert J.


Answers:
author: Shannon S
body: Robert,

Which version of MySQL 5.6 are you using? Note that you cannot use any versions earlier than 5.6.16 per our [Supported Platforms](https://confluence.atlassian.com/conf65/supported-platforms-939702042.html).

The errors you're encountering look much like those documented in this article:

* [Unable to start Confluence due to unsupported version of MySQL](https://confluence.atlassian.com/confkb/unable-to-start-confluence-due-to-unsupported-version-of-mysql-302811898.html)

Regards,  
Shannon

comments: author: Robert Jackson
body: Shannon,

Thank you for reaching out about this. I am using MySQL 5.6.39 with Galera Replication:

```
# mysql --version
mysql Ver 14.14 Distrib 5.6.39, for Linux (x86_64) using EditLine wrapper
# mysql
...
Server version: 5.6.39 MySQL Wsrep Server (GPL), wsrep_25.22
...
```

Has anyone attempted to configure Confluence using MySQL with Galera? High availability is important to me hence why I am performing master-master replication. Ideally, I would just use an RDS instance but I would be unable to make the Confluence optimizations found here:

<https://confluence.atlassian.com/doc/database-setup-for-mysql-128747.html>

At this point, I am willing to try anything. Do you have any recommendations on how I can configure MySQL with master-master replication to work with confluence?

Respectfully,

Robert J.


author: Shannon S
body: Robert,

That I don't know, as I'm not very familiar with Galera and haven't spoken to any customers trying to set it up.

I did find a few things, however:

* [Setup Database Failed](https://community.atlassian.com/t5/Confluence-questions/Setup-Database-Failed/qaq-p/458020)
* [How to Cluster Atlassian Server Apps in your Data Center - JIRA, Confluence, Stash with MySQL Galera Cluster](https://severalnines.com/blog/cluster-atlassian-apps-jira-confluence-stash-mysql-galera-cluster) - this seems to be related to Data Center instead, so I don't know if you were using that.

Can you confirm that you went through the article I sent earlier?

Regards,

Shannon


author: Robert Jackson
body: Shannon,

Thanks for reaching back about this. I did read over the articles that you sent earlier and will look into the new several nines article that you just sent when I have time this evening.

Thank you very much for your time. I will update this ticket with my findings.

Respectfully,

Robert J.