Intermediate confluence 7.4.1 crash
Asked by Pandiyan Muthuraman:

recently production server is up dated to 7.4.1 version.

The confluence service crashes very frequently with below message

ERROR \[http-nio-8090-exec-11\] \[\[Standalone\].\[localhost\].\[/\].\[file-server\]\] log Servlet.service() for servlet \[file-server\] threw exception  
-- url: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg \|  
traceId: 1f5312c88f0c2307 \| userName: anonymous

<br />

ERROR \[http-nio-8090-exec-11\] \[\[Standalone\].\[localhost\].\[/\].\[noop\]\] log Servlet.service() for servlet \[noop\] in context with path \[\] threw exception  
java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg  
at com.atlassian.plugin.webresource.impl.snapshot.Resource$1.writeTo(Resource.java:204)  
at com.atlassian.plugin.webresource.transformer.CompileTimeTransformer$1.writeTo(CompileTimeTransformer.java:59)  
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$8.writeTo(ResourceServingHelpers.java:465)  
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$9.writeTo(ResourceServingHelpers.java:477)  
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$3.writeTo(ResourceServingHelpers.java:264)  
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.lambda$writeTo$0(ResourceServingHelpers.java:187)  
at com.atlassian.plugin.cache.filecache.impl.PassThroughCache.cache(PassThroughCache.java:18)  
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.writeTo(ResourceServingHelpers.java:187)  
at com.atlassian.plugin.webresource.impl.http.Controller.lambda$sendCachedInProduction$0(Controller.java:325)

<br />

--------------------------  
JVM Stats  
--------------------------  
xmx = 4294967296  
usedNonHeap = 504520088  
availableHeap = 2915605528  
availableNonHeap = -1  
allocatedHeap = 2147483648  
freeAllocatedHeap = 768121880  
maxNonHeap = -1  
availablePermGen = 0  
maxPermGen = -1  
maxHeap = 4294967296  
usedHeap = 1379361768  
usedPermGen = -1  
xms = 2147483648  
--------------------------  
Request Information  
--------------------------  
URL: https://dashboard/500page.jsp  
Scheme: https  
Server: dashboard  
Port: 443  
URI: /500page.jsp  
Context Path:  
Servlet Path: /500page.jsp  
Path Info: null  
Query String: null

<br />

--------------------------  
Attributes  
--------------------------  
javax.servlet.forward.request_uri: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg  
javax.servlet.forward.context_path:  
javax.servlet.forward.servlet_path: /s  
javax.servlet.forward.path_info: /-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg  
javax.servlet.forward.mapping: org.apache.catalina.core.ApplicationMapping$MappingImpl@6d5bc8d1  
javax.servlet.error.status_code: 500  
_statichash: -guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9  
com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@317e1276  
com.opensymphony.sitemesh.APPLIED_ONCE: true  
__cleanup_recursion_counter: 0  
javax.servlet.error.message:  
javax.servlet.error.servlet_name: noop  
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true  
atlassian.core.seraph.original.url: /500page.jsp  
com.atlassian.labs.botkiller.BotKillerFilter: true  
com.atlassian.gzipfilter.GzipFilter_already_filtered: true  
Confluence-Request-Time: 1611417825474  
B3-TraceId: 779fa9ec1b773c  
loginfilter.already.filtered: true  
javax.servlet.error.request_uri: /s/-guhr2o/8401/758836457240a7983be3b3db3463da30107b780e/1.0.9/_/download/resources/com.atlassian.confluence.plugins.confluence-icons:confluence-icon-resources/assets/icons/confluence-icon-video.svg  
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true  
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true  
com.atlassian.confluence.web.ConfluenceJohnsonFilter_already_filtered: true  
javax.servlet.error.exception: java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg  
os_securityfilter_already_filtered: true  
com.atlassian.seraph.auth.LoginReason: OUT  
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true  
--------------------------  
Parameters  
--------------------------  
caused by: java.lang.RuntimeException: Cannot read resource assets/icons/confluence-icon-video.svg  
at com.atlassian.plugin.webresource.impl.snapshot.Resource$1.writeTo(Resource.java:204)

2021-01-23 16:04:29,854 WARN \[http-nio-8090-exec-30\] \[common.security.jersey.XsrfResourceFilter\] passesAllXsrfChecks XSRF checks failed for request: https://dashboard/rest/spacetree/1.0/spaces , origin: https://dashboard , referrer: https://dashboard/display/HotSpot/VirtualCalls  
-- referer: https://dashboard/display/HotSpot/VirtualCalls \| url: /rest/spacetree/1.0/spaces \| traceId: 933b5d509e080cf0 \| userName: anonymous


Answers:
author: Gonchik Tsymzhitov
body: Hi [@Pandiyan Muthuraman](/t5/user/viewprofilepage/user-id/4185528) !

Did you clean plugin cache ? <https://confluence.atlassian.com/confkb/how-to-clear-confluence-plugins-cache-297664846.html>

Check the rights ?

And double the logs?

<https://jira.atlassian.com/browse/CONFSERVER-55062>

comments: author: Pandiyan Muthuraman
body: Cleaning plugin cache is workaround and temporary solution. Its happening frequently even after cleaning up.

what was the root cause and how it can be fixed permanently.

Do you think JDK need to be updated ? currently am using jdk 8u271 and solaris x86 machine ( JDK 11 is not supported).


author: Gonchik Tsymzhitov
body: [@Pandiyan Muthuraman](/t5/user/viewprofilepage/user-id/4185528) As I understood, it works after cleanup is that right?

But the problem it happened again ?

Could you check log in catalina.out please?


author: Pandiyan Muthuraman
body: Yes, the issue is happening after cleanup plugin-cache in a day or two. Here is the catalina.out captured during issue.

```
23-Jan-2021 18:32:51.547 WARNING [http-nio-8090-exec-15] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:32:56.614 WARNING [http-nio-8090-exec-6] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:33:16.630 WARNING [http-nio-8090-exec-16] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:33:48.879 WARNING [http-nio-8090-exec-24] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:34:47.663 WARNING [http-nio-8090-exec-24] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://dashboard/rest/spacetree/1.0/spaces, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
23-Jan-2021 18:34:47.931 SEVERE [http-nio-8090-exec-10] com.sun.jersey.spi.container.ContainerResponse.mapException Exception mapper com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper@5e770952 for Throwable com.atlassian.plugins.rest.common.security.AuthenticationRequiredException: Client must be authenticated to access this resource. threw a RuntimeException when attempting to obtain the response





23-Jan-2021 18:34:47.932 SEVERE [http-nio-8090-exec-10] com.sun.jersey.spi.container.ContainerResponse.logException Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: java.text.ParseException: String, es-419, is not a valid language tag
at com.sun.jersey.server.impl.model.HttpHelper.clientError(HttpHelper.java:273)
at com.sun.jersey.server.impl.model.HttpHelper.getAcceptLanguage(HttpHelper.java:228)
at com.sun.jersey.server.impl.VariantSelector.selectVariant(VariantSelector.java:290)
at com.sun.jersey.spi.container.ContainerRequest.selectVariant(ContainerRequest.java:696)
at com.sun.jersey.spi.container.AdaptingContainerRequest.selectVariant(AdaptingContainerRequest.java:286)
at sun.reflect.GeneratedMethodAccessor1564.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.application.WebApplicationImpl$2.invoke(WebApplicationImpl.java:301)
at com.sun.proxy.$Proxy2846.selectVariant(Unknown Source)
at com.atlassian.plugins.rest.common.Status.variantFor(Status.java:304)
at com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper.toResponse(SecurityExceptionMapper.java:22)
at com.atlassian.plugins.rest.common.security.jersey.SecurityExceptionMapper.toResponse(SecurityExceptionMapper.java:16)
at com.sun.jersey.spi.container.ContainerResponse.mapException(ContainerResponse.java:480)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1479)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:162)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:72)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:55)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:43)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:22)
at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:34)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
```


author: Daniel Ebers
body: The error message you have in logs

```
com.sun.jersey.spi.container.ContainerResponse.logException Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: java.text.ParseException: String, es-419, is not a valid language tag
```

might be related to this Bug: <https://jira.atlassian.com/browse/CONFSERVER-59540>  
But I doubt this is the reason why the Confluence instances crashes.  
For me there is nothing obvious visible from logs why I only could refer to a more general guide:  
<https://confluence.atlassian.com/confkb/troubleshooting-confluence-hanging-or-crashing-163415613.html>


author: Pandiyan Muthuraman
body: I have checked atlassian and catalina old logs, where could see below reported issues are occurring for very long time (even before upgrade) and having work around to fix solution. So this couldn't be cause of issue

* not a valid language tag
* Cannot read resource assets/icons/confluence-icon-video.svg .

I have gone through checklist <https://confluence.atlassian.com/confkb/troubleshooting-confluence-hanging-or-crashing-163415613.html> everything looks normal.

I suspect some plugins (atlassian, other vendors) causing issue. Is there any open source / free atlassian plugin to track plugin and generate reports ?


author: Daniel Ebers
body: Not that I would be aware of - but to start with basic things, if you suspect an App being responsible for the crashed you could check if they are up to date. If not, you would need to upgrade them. In case one behaves suspicious you also could check with it's vendor.


author: Gonchik Tsymzhitov
body: [@Pandiyan Muthuraman](/t5/user/viewprofilepage/user-id/4185528) Just curious did you raised a ticket to getsupport.atlassian.com.

Or we need to review all logs from support.zip.

Also, during crash, any problem on GC level ?


Tags: confluence-server

server