Crowd fails with SQL Error: 1205, SQLState: 41000. Any suggestions?
Asked by James Katsis:

Hi guys,  

We use Crowd to authenticate for jira and confluence. We have had this issue about 3 times now which causes an outage. We can easily fix the issue by restarting crowd service and my sql service or simply restart the windows box. We really need a solution to prevent it from occuring again.  

<br />

I have already confirmed that the ransaction-isolation = READ-COMMITTED is in the mysql install directory. We are running crowd 2.04 on Windows server 2003 and mysql version 5.1.47.  

Any help would be appreciated.  

http://confluence.atlassian.com/pages/viewpage.action?pageId=229179847  

..............................................................................................................................................................  

<br />

```
2011-07-25 16:21:53,210 http-8095-90 INFO [integration.directory.connector.ApacheDS15] Performing user search: baseDN = dc=bts,dc=cgu,dc=com,dc=au - filter = (&amp;(objectclass=inetorgperson)(cn=nick.haughton))2011-07-25 16:21:53,866 http-8095-72 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1205, SQLState: 410002011-07-25 16:21:53,866 http-8095-72 ERROR [org.hibernate.util.JDBCExceptionReporter] Lock wait timeout exceeded; try restarting transaction2011-07-25 16:21:53,866 http-8095-72 ERROR [hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with sessionorg.hibernate.exception.GenericJDBCException: Could not execute JDBC batch updateat org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:253)at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:237)at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:142)at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:606)at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709)at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)at $Proxy14.searchPrincipals(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.codehaus.xfire.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:59)at org.codehaus.xfire.service.binding.ServiceInvocationHandler.sendMessage(ServiceInvocationHandler.java:320)at org.codehaus.xfire.service.binding.ServiceInvocationHandler$1.run(ServiceInvocationHandler.java:86)at org.codehaus.xfire.service.binding.ServiceInvocationHandler.execute(ServiceInvocationHandler.java:134)at org.codehaus.xfire.service.binding.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:109)at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)at org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)at org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)at org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)at org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)at org.codehaus.xfire.transport.http.XFireServlet.doPost(XFireServlet.java:116)at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:71)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)at com.atlassian.crowd.console.filter.CrowdOpenSessionInViewFilter.doFilterInternal(CrowdOpenSessionInViewFilter.java:26)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.gzipfilter.GzipFilter.doFilterInternal(GzipFilter.java:75)at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:51)at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)at com.atlassian.crowd.console.filter.CrowdDelegatingFilterProxy.doFilter(CrowdDelegatingFilterProxy.java:38)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)at java.lang.Thread.run(Thread.java:619)Caused by: java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transactionat com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2007)at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1443)at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:246)... 79 more2011-07-25 16:21:53,866 http-8095-72 ERROR [codehaus.xfire.handler.DefaultFaultHandler] Fault occurred!org.springframework.jdbc.UncategorizedSQLException: Hibernate flushing: Could not execute JDBC batch update; uncategorized SQLException for SQL [update cwd_token set directory_id=?, entity_name=?, random_number=?, identifier_hash=?, random_hash=?, created_date=?, last_accessed_date=? where id=?]; SQL state [41000]; error code [1205]; Lock wait timeout exceeded; try restarting transaction; nested exception is java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transactionat org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.translate(SQLStateSQLExceptionTranslator.java:124)at org.springframework.orm.hibernate3.HibernateTransactionManager.convertJdbcAccessException(HibernateTransactionManager.java:750)at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:736)at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:614)at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709)at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)at $Proxy14.searchPrincipals(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.codehaus.xfire.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:59)at org.codehaus.xfire.service.binding.ServiceInvocationHandler.sendMessage(ServiceInvocationHandler.java:320)at org.codehaus.xfire.service.binding.ServiceInvocationHandler$1.run(ServiceInvocationHandler.java:86)at org.codehaus.xfire.service.binding.ServiceInvocationHandler.execute(ServiceInvocationHandler.java:134)at org.codehaus.xfire.service.binding.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:109)at org.codehaus.xfire.handler.HandlerPipeline.invoke(HandlerPipeline.java:131)at org.codehaus.xfire.transport.DefaultEndpoint.onReceive(DefaultEndpoint.java:64)at org.codehaus.xfire.transport.AbstractChannel.receive(AbstractChannel.java:38)at org.codehaus.xfire.transport.http.XFireServletController.invoke(XFireServletController.java:304)at org.codehaus.xfire.transport.http.XFireServletController.doService(XFireServletController.java:129)at org.codehaus.xfire.transport.http.XFireServlet.doPost(XFireServlet.java:116)at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:71)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198)at com.atlassian.crowd.console.filter.CrowdOpenSessionInViewFilter.doFilterInternal(CrowdOpenSessionInViewFilter.java:26)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.gzipfilter.GzipFilter.doFilterInternal(GzipFilter.java:75)at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:51)at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)at com.atlassian.crowd.console.filter.CrowdDelegatingFilterProxy.doFilter(CrowdDelegatingFilterProxy.java:38)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:41)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)at java.lang.Thread.run(Thread.java:619)
```


Answers:
author: Colin Goudie
body: What is the your connection url setting in crowd.cfg.xml? This is in the your crowd home directory.

Does it have the autoReconnect parameter?

comments: author: James Katsis
body: Hi Colin, Yes it does. the following line exists in the crowd.cfg.xml

\<property name="hibernate.connection.url"\>jdbc:mysql://localhost/crowd?autoReconnect=true\&amp;characterEncoding=utf8\&amp;useUnicode=true\<


author: James Katsis
body: Hi Colin, We originally setup the my sql connection using the JIRA configuration tool, rather than the manual method. This means there is no dbconfig.xml in the jira home directory. Where are the settings stored? How can i tell if we have the \<tt\>\<validation-query\>select 1\</validation-query\>\</tt\> line in the database config? Can i just create a dbconfig.xml?


author: Colin Goudie
body: My first thought is that the jdbc settings are not using a validation query to keep the MySQL connection around.

See this page in the JIRA docs, it may lead you to the answer

<http://confluence.atlassian.com/display/JIRA/Connecting+JIRA+to+MySQL#ConnectingJIRAtoMySQL-4.2ConnectingJIRAtoMySQLManually>