Why does full synchronization between Crowd and Confluence fail ?
Asked by Erkki_Aalto:

We have a problem, fortunately with a test system, with synchronization of Confluence users with Crowd server.

Full synchronization fails every time. I have a feelling that the problem is in the Crowd side.

Confluence is version 5.8.4

Crowd is version 2.8.0

The Crowd server log has errors like:

2015-09-16 08:52:34,846 ajp-bio-8009-exec-6 DEBUG \[plugin.rest.filter.BasicApplicationAuthenticationFilter\] Application 'sikawiki' is already authenticated  
2015-09-16 08:52:34,848 ajp-bio-8009-exec-6 DEBUG \[crowd.console.filter.CrowdOpenSessionInViewFilter\] Closing Hibernate Session in OpenSessionInViewFilter  
2015-09-16 08:52:49,428 ajp-bio-8009-exec-13 ERROR \[server.impl.application.WebApplicationImpl\] The response of the WebApplicationException cannot be utilized as the response is already committed. Re-throwing to the HTTP container  
[javax.ws](http://javax.ws).rs.WebApplicationException: com.ctc.wstx.exc.WstxIOException: null  
at com.atlassian.crowd.plugin.rest.service.controller.MembershipsController$OutputAsXml.write(MembershipsController.java:234)  
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)  
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)  
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)  
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)  
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)  
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)  
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)  
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)  
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:178)  
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)  
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:73)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:26)  
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:40)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:186)  
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)  
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)  
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)  
at com.atlassian.crowd.console.filter.CrowdDelegatingFilterProxy.doFilter(CrowdDelegatingFilterProxy.java:39)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:98)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:40)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.crowd.plugin.rest.filter.SeraphDisablerFilter.doFilter(SeraphDisablerFilter.java:33)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.crowd.plugin.rest.filter.BasicApplicationAuthenticationFilter.doFilter(BasicApplicationAuthenticationFilter.java:97)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:151)  
at com.atlassian.crowd.console.filter.CrowdOpenSessionInViewFilter.doFilterInternal(CrowdOpenSessionInViewFilter.java:26)  
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.crowd.plugin.web.filter.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:31)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.crowd.plugin.rest.filter.RestServiceVersionFilter.doFilter(RestServiceVersionFilter.java:86)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)  
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.crowd.plugin.web.filter.HttpContextFilter.doFilter(HttpContextFilter.java:34)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)  
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)  
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)  
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)  
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)  
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)  
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)  
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)  
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)  
at [org.apache.tomcat.util.net](http://org.apache.tomcat.util.net).JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: com.ctc.wstx.exc.WstxIOException: null  
at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteEndTag(BaseNsStreamWriter.java:671)  
at com.ctc.wstx.sw.BaseNsStreamWriter.writeEndElement(BaseNsStreamWriter.java:282)  
at com.atlassian.crowd.plugin.rest.service.controller.MembershipsController.writeXmlToStream(MembershipsController.java:118)  
at com.atlassian.crowd.plugin.rest.service.controller.MembershipsController$OutputAsXml.write(MembershipsController.java:230)  
... 105 more  
Caused by: ClientAbortException: [java.net](http://java.net).SocketException: Broken pipe  
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)  
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)  
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)  
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:438)  
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:426)  
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)  
at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:307)  
at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:134)  
at [com.ctc.wstx.io](http://com.ctc.wstx.io).UTF8Writer.write(UTF8Writer.java:139)  
at com.ctc.wstx.sw.BufferingXmlWriter.flushBuffer(BufferingXmlWriter.java:1103)  
at com.ctc.wstx.sw.BufferingXmlWriter.fastWriteRaw(BufferingXmlWriter.java:1149)  
at com.ctc.wstx.sw.BufferingXmlWriter.writeEndTag(BufferingXmlWriter.java:771)  
at com.ctc.wstx.sw.BufferingXmlWriter.writeEndTag(BufferingXmlWriter.java:789)  
at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteEndTag(BaseNsStreamWriter.java:669)  
... 108 more  
Caused by: [java.net](http://java.net).SocketException: Broken pipe  
at [java.net](http://java.net).SocketOutputStream.socketWrite0(Native Method)  
at [java.net](http://java.net).SocketOutputStream.socketWrite(SocketOutputStream.java:113)  
at [java.net](http://java.net).SocketOutputStream.write(SocketOutputStream.java:159)  
at org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:295)  
at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1197)  
at org.apache.coyote.Response.doWrite(Response.java:520)  
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)  
... 121 more  
2015-09-16 08:52:49,429 ajp-bio-8009-exec-13 DEBUG \[crowd.console.filter.CrowdOpenSessionInViewFilter\] Closing Hibernate Session in OpenSessionInViewFilter  
2015-09-16 08:52:49,475 ajp-bio-8009-exec-13 ERROR \[\[Catalina\].\[localhost\].\[/crowd\].\[default\]\] Servlet.service() for servlet \[default\] in context with path \[/crowd\] threw exception  
[javax.ws](http://javax.ws).rs.WebApplicationException: com.ctc.wstx.exc.WstxIOException: null  
at com.atlassian.crowd.plugin.rest.service.controller.MembershipsController$OutputAsXml.write(MembershipsController.java:234)  
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)  
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)  
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)  
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)  
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)  
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)  
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)  
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)  
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:178)  
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)  
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:73)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:26)  
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:40)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:186)  
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)  
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)  
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)  
at com.atlassian.crowd.console.filter.CrowdDelegatingFilterProxy.doFilter(CrowdDelegatingFilterProxy.java:39)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:98)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:40)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.crowd.plugin.rest.filter.SeraphDisablerFilter.doFilter(SeraphDisablerFilter.java:33)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.crowd.plugin.rest.filter.BasicApplicationAuthenticationFilter.doFilter(BasicApplicationAuthenticationFilter.java:97)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:151)  
at com.atlassian.crowd.console.filter.CrowdOpenSessionInViewFilter.doFilterInternal(CrowdOpenSessionInViewFilter.java:26)  
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.crowd.plugin.web.filter.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:31)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)  
at com.atlassian.crowd.plugin.rest.filter.RestServiceVersionFilter.doFilter(RestServiceVersionFilter.java:86)  
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)  
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)  
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)  
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at com.atlassian.crowd.plugin.web.filter.HttpContextFilter.doFilter(HttpContextFilter.java:34)  
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)  
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)  
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)  
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)  
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)  
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)  
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)  
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)  
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)  
at [org.apache.tomcat.util.net](http://org.apache.tomcat.util.net).JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: com.ctc.wstx.exc.WstxIOException: null  
at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteEndTag(BaseNsStreamWriter.java:671)  
at com.ctc.wstx.sw.BaseNsStreamWriter.writeEndElement(BaseNsStreamWriter.java:282)  
at com.atlassian.crowd.plugin.rest.service.controller.MembershipsController.writeXmlToStream(MembershipsController.java:118)  
at com.atlassian.crowd.plugin.rest.service.controller.MembershipsController$OutputAsXml.write(MembershipsController.java:230)  
... 105 more  
Caused by: ClientAbortException: [java.net](http://java.net).SocketException: Broken pipe  
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)  
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)  
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)  
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:438)  
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:426)  
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)  
at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:307)  
at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:134)  
at [com.ctc.wstx.io](http://com.ctc.wstx.io).UTF8Writer.write(UTF8Writer.java:139)  
at com.ctc.wstx.sw.BufferingXmlWriter.flushBuffer(BufferingXmlWriter.java:1103)  
at com.ctc.wstx.sw.BufferingXmlWriter.fastWriteRaw(BufferingXmlWriter.java:1149)  
at com.ctc.wstx.sw.BufferingXmlWriter.writeEndTag(BufferingXmlWriter.java:771)  
at com.ctc.wstx.sw.BufferingXmlWriter.writeEndTag(BufferingXmlWriter.java:789)  
at com.ctc.wstx.sw.BaseNsStreamWriter.doWriteEndTag(BaseNsStreamWriter.java:669)  
... 108 more  
Caused by: [java.net](http://java.net).SocketException: Broken pipe  
at [java.net](http://java.net).SocketOutputStream.socketWrite0(Native Method)  
at [java.net](http://java.net).SocketOutputStream.socketWrite(SocketOutputStream.java:113)  
at [java.net](http://java.net).SocketOutputStream.write(SocketOutputStream.java:159)  
at org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:295)  
at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1197)  
at org.apache.coyote.Response.doWrite(Response.java:520)  
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)  
... 121 more  
2015-09-16 08:52:53,792 ajp-bio-8009-exec-8 DEBUG \[crowd.console.filter.CrowdOpenSessionInViewFilter\] Using SessionFactory 'sessionFactory' for OpenSessionInViewFilter

The Confluence log has errors like:

2015-09-16 08:52:46,417 ERROR \[scheduler_Worker-2\] \[atlassian.crowd.directory.DbCachingDirectoryPoller\] pollChanges Error occurred while refreshing the cache for directory \[ 166952961 \].  
com.atlassian.crowd.exception.OperationFailedException: javax.xml.bind.UnmarshalException  
- with linked exception:  
\[org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected\]  
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:497)  
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getMemberships(RestCrowdClient.java:914)  
at com.atlassian.crowd.directory.RemoteCrowdDirectory.getMemberships(RemoteCrowdDirectory.java:1068)  
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.getMemberships(AbstractCacheRefresher.java:107)  
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseMemberships(AbstractCacheRefresher.java:185)  
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:98)  
at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:67)  
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1122)  
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:76)  
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)  
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:93)  
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:135)  
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:101)  
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:80)  
at com.atlassian.scheduler.quartz1.Quartz1Job.execute(Quartz1Job.java:32)  
at org.quartz.core.JobRunShell.run(JobRunShell.java:223)  
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool.lambda$runInThread$46(ConfluenceQuartzThreadPool.java:19)  
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$$Lambda$89/97766329.run(Unknown Source)  
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)  
Caused by: javax.xml.bind.UnmarshalException  
- with linked exception:  
\[org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected\]  
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:213)  
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:190)  
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:241)  
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:489)  
... 18 more  
Caused by: org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected  
at [org.apache.http.impl.io](http://org.apache.http.impl.io).ChunkedInputStream.getChunkSize(ChunkedInputStream.java:268)  
at [org.apache.http.impl.io](http://org.apache.http.impl.io).ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)  
at [org.apache.http.impl.io](http://org.apache.http.impl.io).ChunkedInputStream.read(ChunkedInputStream.java:186)  
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)  
at [java.io](http://java.io).SequenceInputStream.read(SequenceInputStream.java:207)  
at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)  
at [org.apache.xerces.impl.io](http://org.apache.xerces.impl.io).UTF8Reader.read(Unknown Source)  
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)  
at org.apache.xerces.impl.XMLEntityScanner.scanQName(Unknown Source)  
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)  
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)  
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)  
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)  
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)  
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)  
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)  
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)  
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:211)  
... 21 more


Answers:
author: Erkki_Aalto
body: Now I realized one difference between our test and production systems. The test system had a great number of inactive user accounts (tens of thousands!). When I deleted the inactive accounts, full synchronization is succeeding again. Clearly there is some problem in very large number of user accounts.


author: lavitz slambert
body: You are more than right in aiming for the proxy IMO :smile:, I saw that problem in another post as well:

<https://answers.atlassian.com/questions/259235>

Lastly can you check the crowd base url at the crowd.properties file is using http instead of https? Usually when we have a proxy in place folks uses https on proxies and when crowd tries to communicate using http the redirection the proxy does (from http to https) causes this kind of error as you can see below:

<https://answers.atlassian.com/questions/117748>

Hope this give you a direction.  
Thanks and Regards,  
Lavitz - the guy that uses Confluence on Windows as a Hobby!


author: Mallmann
body: Hi Erkki,

The error "ClientAbortException: [java.net](http://java.net/).SocketException: Broken pipe" from your logs seems to point it to be a network related problem.

It seems that your connection is getting closed while the synchronisation is still on going:
> org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected

Maybe it's a timeout issue, can you confirm if you are using any kind of proxy in between these two applications?

Also, does it happen if you added a fresh Crowd directory in your Confluence? What about creating a new Confluence application inside Crowd and map only a few users in there?

I hope it helps!

Eduardo


author: Erkki_Aalto
body: No, I don't think it is a network problem. SSL error log has the messages

\[Thu Sep 17 09:27:13.716815 2015\] \[proxy_ajp:error\] \[pid 5716\] \[client 128.214.222.54:43125\] AH00992: ajp_read_header: ajp_ilink_receive failed

\[Thu Sep 17 09:27:13.716833 2015\] \[proxy_ajp:error\] \[pid 5716\] (70007)The timeout specified has expired: \[client 128.214.222.54:43125\] AH00893: dialog to (null) (\*) failed

It seems that Crowd has stopped talking to the proxy. To make things more mysterious, one of the test Confluence systems performed successfull full synchronization when I enabled profiling in the Crowd server but I could not repeat that in another system. There are also no problems in our production systems that are really in heavy use.


Tags: confluence-administration

crowd