Cannot rebuild indexes, Hibernate error
Asked by joshua kugler:

We're in the process of upgrading from Confluence 3.3 to Confluence 4.2.13. We first did the upgrade to 3.5, and that went well. The upgrade to 4.2.13 is working, but the indexes are not rebuilding. The error int he log is below.

Java 1.6.0_34

MySQL 5.5.25

Java Mysql Connector 5.1.21

This is the search index. The "Did you mean" index rebuilds with no issue. Everything else with the site seems to be working. Any ideas?

```
2012-08-27 17:48:52,540 WARN [confluence-interactive-reindexing-thread] [v2.c3p0.impl.DefaultConnectionTester] statusOnException SQL State '08007' of Exception tested by statusOnException() implies that the database is invalid, and the pool should refill itself with fresh Connections. -- referer: http://wiki-up.lab.whamcloud.com/admin/search-indexes.action | url: /admin/reindex.action | userName: joshua | action: reindexcom.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown.	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)	at java.lang.reflect.Constructor.newInstance(Unknown Source)	at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)	at com.mysql.jdbc.Util.getInstance(Util.java:381)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1012)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:986)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:981)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)	at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1660)	at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803)	at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63)	at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)	at $Proxy87.indexEntities(Unknown Source)	at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:35)	at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:26)	at java.lang.Thread.run(Unknown Source)2012-08-27 17:48:52,544 WARN [confluence-interactive-reindexing-thread] [v2.c3p0.impl.C3P0PooledConnectionPool] doMarkPoolStatus A ConnectionTest has failed, reporting that all previously acquired Connections are likely invalid. The pool will be reset. -- referer: http://wiki-up.lab.whamcloud.com/admin/search-indexes.action | url: /admin/reindex.action | userName: joshua | action: reindex2012-08-27 17:48:52,545 ERROR [confluence-interactive-reindexing-thread] [sf.hibernate.transaction.JDBCTransaction] commit Commit failed -- referer: http://wiki-up.lab.whamcloud.com/admin/search-indexes.action | url: /admin/reindex.action | userName: joshua | action: reindexcom.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown.	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)	at java.lang.reflect.Constructor.newInstance(Unknown Source)	at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)	at com.mysql.jdbc.Util.getInstance(Util.java:381)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1012)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:986)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:981)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)	at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1660)	at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803)	at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63)	at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)	at $Proxy87.indexEntities(Unknown Source)	at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:35)	at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:26)	at java.lang.Thread.run(Unknown Source)2012-08-27 17:48:52,549 ERROR [confluence-interactive-reindexing-thread] [confluence.util.longrunning.ConfluenceAbstractLongRunningTask] run Long running task "Rebuilding Search Index" failed to run. -- referer: http://wiki-up.lab.whamcloud.com/admin/search-indexes.action | url: /admin/reindex.action | userName: joshua | action: reindexorg.springframework.transaction.TransactionSystemException: Could not commit Hibernate transaction; nested exception is net.sf.hibernate.TransactionException: Commit failed with SQL exception: 	at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:514)	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)	at $Proxy87.indexEntities(Unknown Source)	at com.atlassian.confluence.search.IndexingTask.runInternal(IndexingTask.java:35)	at com.atlassian.confluence.util.longrunning.ConfluenceAbstractLongRunningTask.run(ConfluenceAbstractLongRunningTask.java:26)	at java.lang.Thread.run(Unknown Source)Caused by: net.sf.hibernate.TransactionException: Commit failed with SQL exception: 	at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:71)	at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)	... 10 moreCaused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown.	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)	at java.lang.reflect.Constructor.newInstance(Unknown Source)	at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)	at com.mysql.jdbc.Util.getInstance(Util.java:381)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1012)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:986)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:981)	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)	at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1660)	at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803)	at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:63)	... 11 more
```


Answers:
author: Harry Chan
body: Hi, please try to add ?autoReconnect=true to the end of your database's JDBC URL and add the validationQuery parameter.

Follow this guide here <https://confluence.atlassian.com/pages/viewpage.action?pageId=104300792> for more information.

comments: author: joshua kugler
body: I don't seem to have a config file as referenced by the above page. I have \<HOMEDIR\>/conf/confluence.cfg.xml. I'll paste it below. From the URL you pasted, I'm not 100% sure how to add the validations query. And I already do have the autoreconnect=true in the URL. Snipping the irrelevant parts.

```
&lt;?xml version="1.0" encoding="UTF-8"?&gt;&lt;confluence-configuration&gt;    &lt;property name="hibernate.c3p0.acquire_increment"&gt;1&lt;/property&gt;    &lt;property name="hibernate.c3p0.idle_test_period"&gt;100&lt;/property&gt;    &lt;property name="hibernate.c3p0.max_size"&gt;30&lt;/property&gt;    &lt;property name="hibernate.c3p0.max_statements"&gt;0&lt;/property&gt;    &lt;property name="hibernate.c3p0.min_size"&gt;0&lt;/property&gt;    &lt;property name="hibernate.c3p0.timeout"&gt;30&lt;/property&gt;    &lt;property name="hibernate.connection.driver_class"&gt;com.mysql.jdbc.Driver&lt;/property&gt;    &lt;property name="hibernate.connection.password"&gt;...&lt;/property&gt;    &lt;property name="hibernate.connection.url"&gt;jdbc:mysql://localhost/confluence?autoReconnect=true&amp;amp;sessionVariables=storage_engine%3DInnoDB&lt;/property&gt;    &lt;property name="hibernate.connection.username"&gt;confluence&lt;/property&gt;    &lt;property name="hibernate.database.lower_non_ascii_supported"&gt;true&lt;/property&gt;    &lt;property name="hibernate.dialect"&gt;com.atlassian.hibernate.dialect.MySQLDialect&lt;/property&gt;    &lt;property name="hibernate.setup"&gt;true&lt;/property&gt;  &lt;/properties&gt;&lt;/confluence-configuration&gt;
```


author: joshua kugler
body: Found it here: <https://confluence.atlassian.com/display/DOC/Surviving+Database+Connection+Closures> Will add it, and report back.


author: joshua kugler
body: Tried rebuilding the index again, and had the same issue. This was after adding the validation query, and restarting confluence.


author: Harry Chan
body: Can you try to increase the lock wait_timeout on MySQL server? Instructions on how to do it can be found here <https://confluence.atlassian.com/display/JIRAKB/Reindexing+JIRA+Results+in+Communications+Link+Failure+During+Commit>

If that doesn't fix it, try to increase the global interactive timeout as documented here https://confluence.atlassian.com/display/CONFKB/Attachment+Upload+Failed+with+'Communications+link+failure+during+commit()'+Error.

It's definitely timing out before it can commit. That is the problem.


author: Harry Chan
body: Did you also add ?autoReconnect=true to the end of the JDBC connection URL?


author: joshua kugler
body: As shown in the config file pasted above, I already had that in the URL.


author: joshua kugler
body: That was it. Thanks. It was 180, and I increased it to 600.

Thank you!