Confluence Server: StuckThread after Upgrade to 7.0.1
Asked by Trevor Downing:

We run Confluence Server on 2012 R2 and it was running smoothly until we upgraded from 6.15.8 to 7.0.1.

After the upgrade we encountered the following issues:

Resolved: The top bar was failing to load and in it's place was code. We followed an article about reloading plugins and it resolved the issue for us.

<https://confluence.atlassian.com/confkb/how-to-reset-all-confluence-plugins-back-to-their-default-state-through-the-database-859509873.html>

Unresolved: when Confluence was started back up and I was at the login screen, everything seemed to be running smoothly. I could browser our public articles and there would be no irregular delays loading the page. However, after I logged in, I noticed significant delay in navigating from page to page (over a minute each time).

I started looking around in the logs and found a few different logs but I was unable to ascertain what the connection between the logs are or what underlying issue is causing the delays.

Here are some of the logs:

From {service-name}-stderr.{today's-date}.log in confluence\\logs\\

org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread \[https-jsse-nio2-8443-exec-1\] (id=\[15\]) has been active for \[62,062\] milliseconds (since \[9/30/19 8:46 AM\]) to serve the same request for \[{base-url}\] and may be stuck (configured threshold for this StuckThreadDetectionValve is \[60\] seconds). There is/are \[1\] thread(s) in total that are monitored by this Valve and may be stuck.

org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread \[https-jsse-nio2-8443-exec-1\] (id=\[15\]) was previously reported to be stuck but has completed. It was active for approximately \[360,832\] milliseconds.

From Confluence\\Application Data\\Confluence\\Logs\\atlassian-confluence.log:

ERROR \[https-jsse-nio2-8443-exec-9\] \[renderer.internal.http.HttpClientFetcher\] fetch Unable to perform a request to: {base-url}/rest/gadgets/1.0/g/messagebundle/en_US/gadget.common%2Cgadget.confluence  
-- referer: {base-URL}/display/KB/{article-name} \| url: /plugins/macrobrowser/browse-macros.action \| traceId: ---- \| userName: ----- \| action: browse-macros  
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

\[oauth.serviceprovider.internal.AuthenticatorImpl\] logOAuthProblem Problem encountered authenticating OAuth client for url "{base-URL}/rest/knowledge-base/1.0/search/KB", error was "consumer_key_unknown", with parameters "{oauth_problem=consumer_key_unknown}"

I'm unsure which of these error logs is most significant and I'm not sure where to go from here. I appreciate advice you have to offer!


Answers:
author: Trevor Downing
body: Update on this: we rolled the server / DB back to a 6.x version that was working for us. I'm still searching for advice because we want to be able to upgrade, but we don't know what was causing the stuck thread.

Thank you for anyone who offers advice on this.


author: Justin Parker
body: I get those errors also and I am on Confluence 7.0.1 (upgraded from 6.x) and below is my env info. Not sure if you are running the same minus the Linux OS in VMware Fusion.

Have you tried running a diagnostic and look in the /var/atlassian/application-data/confluence/logs/atlassian-diagnostics.log file?

> grep -i stuck /var/atlassian/application-data/confluence/logs/atlassian-confluence.log -c  
> 11 counts
>
> uname -a  
> Linux localhost.localdomain 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> rpm -qa \| grep mysql
>
> mysql-community-common-5.7.27-1.el7.x86_64  
> mysql-community-client-5.7.27-1.el7.x86_64  
> mysql-community-libs-compat-5.7.27-1.el7.x86_64  
> mysql-community-libs-5.7.27-1.el7.x86_64  
> mysql-community-server-5.7.27-1.el7.x86_64
>
> cat /etc/os-release  
> NAME="CentOS Linux"  
> VERSION="7 (Core)"  
> ID="centos"  
> ID_LIKE="rhel fedora"  
> VERSION_ID="7"  
> PRETTY_NAME="CentOS Linux 7 (Core)"  
> ANSI_COLOR="0;31"  
> CPE_NAME="cpe:/o:centos:centos:7"  
> HOME_URL="<https://www.centos.org/>"  
> BUG_REPORT_URL="<https://bugs.centos.org/>"
>
> CENTOS_MANTISBT_PROJECT="CentOS-7"  
> CENTOS_MANTISBT_PROJECT_VERSION="7"  
> REDHAT_SUPPORT_PRODUCT="centos"  
> REDHAT_SUPPORT_PRODUCT_VERSION="7"
>
> <br />
>
> cat /proc/cpuinfo
>
> rocessor : 1  
> vendor_id : GenuineIntel  
> cpu family : 6  
> model : 94  
> model name : Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz  
> stepping : 3  
> microcode : 0xcc  
> cpu MHz : 2591.775  
> cache size : 6144 KB
>
> cat /proc/meminfo
>
> MemTotal: 1863236 kB  
> MemFree: 68260 kB  
> MemAvailable: 22876 kB  
> Buffers: 1496 kB  
> Cached: 28068 kB  
> SwapCached: 539416 kB  
> Active: 1236028 kB  
> Inactive: 426012 kB  
> Active(anon): 1220520 kB  
> Inactive(anon): 412600 kB  
> Active(file): 15508 kB  
> Inactive(file): 13412 kB  
> Unevictable: 0 kB  
> Mlocked: 0 kB  
> SwapTotal: 3905532 kB  
> SwapFree: 2399740 kB  
> Dirty: 28 kB  
> Writeback: 0 kB  
> AnonPages: 1606888 kB  
> Mapped: 14456 kB  
> Shmem: 660 kB  
> Slab: 67904 kB  
> SReclaimable: 29868 kB  
> SUnreclaim: 38036 kB  
> KernelStack: 8368 kB  
> PageTables: 9892 kB  
> NFS_Unstable: 0 kB  
> Bounce: 0 kB  
> WritebackTmp: 0 kB  
> CommitLimit: 4837148 kB  
> Committed_AS: 3218280 kB  
> VmallocTotal: 34359738367 kB  
> VmallocUsed: 179620 kB  
> VmallocChunk: 34359310332 kB  
> HardwareCorrupted: 0 kB  
> AnonHugePages: 270336 kB  
> CmaTotal: 0 kB  
> CmaFree: 0 kB  
> HugePages_Total: 0  
> HugePages_Free: 0  
> HugePages_Rsvd: 0  
> HugePages_Surp: 0  
> Hugepagesize: 2048 kB  
> DirectMap4k: 75648 kB  
> DirectMap2M: 2021376 kB  
> DirectMap1G: 0 kB

comments: author: Trevor Downing
body: Hi Parker,

Thank you for your response. I haven't ran the diagnostics, I'm not sure how and it is very difficult to navigate to anything inside the web-service right now because it won't load half the time and if it does it may take multiple minutes.

I tried to go to the troubleshooting tools section and there was a popup on the right side of the screen that said:  

"Your URL doesn't match

Confluence's base URL is set to "{base-url}:{port}" but you are accessing Confluence from "{base-url}:{port}undefined"

It's the exact same url with the word "undefined" at the end. I don't know if this is relevant but I thought I would share it anyways.

In Troubleshooting and Support Tools it shows the Instance health checks and everything has a passing checkmark. I will try to find the diagnostics you mentioned running.


author: Trevor Downing
body: I was reading some more and I found the database latency for me is very high.

Under System Information it says: Database Latency: 48ms

I read this should be below 2ms so I'm looking for ways to troubleshoot this.


author: Trevor Downing
body: Follow up: The DB Latency metric usually appears low 0-10 but I occasionally find it higher like when I found it at 48ms.

I also found the following two entries in the logs so I'm wondering if this could be related:

02-Oct-2019 10:28:16.515 WARNING \[Catalina-utility-2\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application \[ROOT\] registered the JDBC driver \[com.microsoft.sqlserver.jdbc.SQLServerDriver\] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

02-Oct-2019 10:28:16.530 WARNING \[Catalina-utility-2\] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application \[ROOT\] appears to have started a thread named \[C3P0PooledConnectionPoolManager\[identityToken-\>TOKEN  
HelperThread-#0\] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:  
java.lang.Object.wait(Native Method)  
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

02-Oct-2019 10:32:23.197 SEVERE \[https-jsse-nio2-8443-exec-6\] org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor.failed Socket accept failed  
java.io.IOException: The specified network name is no longer available.

I'm wondering if these offer any insight about the issue I'm facing...  
I ran a thread dump but I'm not sure how to interpret the results or what to look for.

Thank you for any help!


author: Justin Parker
body: What database and version are you running? What JDBC driver do you have running?

Is the database running on the same server as Confluence?


author: Trevor Downing
body: Microsoft SQL Server

12.00.5687

Driver version: 6.3.0.0

The DB is running on a different server than Confluence.


Tags: confluence-server

server

thread