Everyday we are receiving jira High CPU Usage Alert
Asked by Shubham Purohit:

Hi there   
Everyday we are receiving High CPU Usage alert around 1:30 Am and auto resolved at 2:00 Am.  
We have checked the processes which are running over the tlpajira-1/2 server and found the culprit process **Java** running with user "appjira  
*PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND*   
*56245 appjira 20 0 21.5g 16.7g 1.4g S 376.3 53.9 9254:20 java*   
*57816 p228973 20 0 172908 2428 1632 R 4.7 0.0 0:03.70 top*   
*4077 splunk 20 0 596072 247892 12740 S 1.6 0.8 603:39.82 splunkd*  
Checked the Parent process with the PID. and found some issue related to the memory allocation and garbage collection, but after checking the **vim atlassian-jira-gc-2024-06-03_22-55-37.log.3.current** file found the GC process is running and completing successfully within relatively short pauses, which proves GC is not primary cause. also checked the jira application logs and found error related to plugins module.

***vim access_log.2024-06-13***   
*172.21.255.237 15x9575031x1 - \[13/Jun/2024:00:15:51 -0500\] "**GET /s/d292f865eed19dadcf6169462458d65b-CDN/jjsogp/940008/1dlckms/9.4.8/_/download/batch/com.atlassian.jira.jira-project-config-plugin:error-module/com.atlassian.jira.jira-project-config-plugin:error-module.js**?locale=en-US HTTP/1.1" 200 1469 15 "* [*https://jira.edwardjones.com/projects/CSSV?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page"*](https://jira.edwardjones.com/projects/CSSV?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page%22)*"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0" "p67vq7"*Under the**catalina.2024-06-13.log** an issue was identified where a thread in the application was stuck and had been active for an extended period of time (over 124 seconds) and A new log entry indicated that this previous stuck thread eventually completed its task. From Grafana we verified after completion of the task the CPU get down at its normal state. there are multiple stuck threads under log file.  

<br />

***vim catalina.2024-06-13.log:***   
***18-Jun-2024 09:07:16.085** WARNING \[Catalina-utility-1\] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread \[https-jsse-nio-8443-exec-320\] (id=\[249734\]) has been active for \[129,490\] milliseconds (since \[6/18/24 9:05 AM\]) to serve the same request for \[* [*https://jira.edwardjones.com/rest/api/latest/search?jql=project+%3D+%22QE+Shared+Services%22+AND+status+not+in+%28Canceled%2C+Cancel%2C+Cancelled%29+AND+type+%3D+Bug+AND+createdDate+%3E%3D+%222024%2F01%2F22%22+AND+cf%5B10502%5D+in+%28+%22CLONE+-+Pre-Prod%22%2C+%22clone+-+UAT%22%29+and+%22Epic+Link%22+%3D+ENFWKS-23513+ORDER+BY++created+DESC\&startAt=0\&maxResults=500\&expand=changelog\]*](https://jira.edwardjones.com/rest/api/latest/search?jql=project+%3D+%22QE+Shared+Services%22+AND+status+not+in+%28Canceled%2C+Cancel%2C+Cancelled%29+AND+type+%3D+Bug+AND+createdDate+%3E%3D+%222024%2F01%2F22%22+AND+cf%5B10502%5D+in+%28+%22CLONE+-+Pre-Prod%22%2C+%22clone+-+UAT%22%29+and+%22Epic+Link%22+%3D+ENFWKS-23513+ORDER+BY++created+DESC&startAt=0&maxResults=500&expand=changelog])*and may be stuck (configured threshold for this StuckThreadDetectionValve is \[120\] seconds). There is/are \[1\] thread(s) in total that are monitored by this Valve and may be stuck.
<br />***18-Jun-2024 09:07:46.120** WARNING \[Catalina-utility-2\] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread \[https-jsse-nio-8443-exec-320\] (id=\[249734\]) was previously reported to be stuck but has completed. It was active for approximately \[152,477\] milliseconds

Could you please guide us how to resolve this issue. as it is recurring every day


Answers:
author: Jim Knepley - ReleaseTEAM
body: Maybe traffic between your servers and the IP address(es) behind jira.edwardjones.com is being blocked/dropped?

I've seen stuck threads running up the CPU if Jira isn't able to connect to an Internet-facing website. If Jira's attempts to connect to the Internet are silently dropped for whatever reason (egress firewall in my case), that could be the root of the issue.


author: Rob Horan
body: Every day off hours sounds like a recurring backup or indexing operation.


Tags: data-center

jira-data-center