Very Poor Performance of JIRA Report Macros
Asked by Chad Barnes:

We are seeing very poor performance of our JIRA report macros. Specificaly, the JIRA Report blueprint-created pages. When we refresh these pages, it takes a very long time (30-60 seconds) to load. While loading, we observe 4 "JIRA macros executor" threads running. In fact, these threads have been "running" longer than they have been "waiting". Sometimes these pages simply timeout.

Our production JIRA server is not exceptionally busy AND the same filters that the Confluence report page uses return very quickly in JIRA.

Any ideas what could be causing such poor performance in the Confluence pages that access JIRA content?

Here is a thread dump:

```
2014-11-25 09:25:32"Jira macros executor:thread-1" - Thread t@395   java.lang.Thread.State: RUNNABLE	at java.math.BigInteger.mulAdd(BigInteger.java:1935)	at java.math.BigInteger.montReduce(BigInteger.java:1880)	at java.math.BigInteger.oddModPow(BigInteger.java:1850)	at java.math.BigInteger.modPow(BigInteger.java:1599)	at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)	at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)	at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)	at javax.crypto.Cipher.doFinal(Cipher.java:2087)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)	- locked &lt;a89a00a&gt; (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)	at com.google.common.cache.LocalCache.get(LocalCache.java:3970)	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)	at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)	at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXML(DefaultJiraIssuesManager.java:90)	at com.atlassian.confluence.extra.jira.CacheJiraIssuesManager.retrieveXML(CacheJiraIssuesManager.java:41)	at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXMLAsChannel(DefaultJiraIssuesManager.java:172)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.populateContextMapForStaticCountIssues(JiraIssuesMacro.java:907)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.createContextMapFromParams(JiraIssuesMacro.java:495)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.execute(JiraIssuesMacro.java:1195)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)	at java.util.concurrent.FutureTask.run(FutureTask.java:262)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)	at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:	- locked &lt;5189a5d5&gt; (a java.util.concurrent.ThreadPoolExecutor$Worker)"Jira macros executor:thread-2" - Thread t@396   java.lang.Thread.State: RUNNABLE	at java.math.BigInteger.mulAdd(BigInteger.java:1930)	at java.math.BigInteger.squareToLen(BigInteger.java:1319)	at java.math.BigInteger.oddModPow(BigInteger.java:1849)	at java.math.BigInteger.modPow(BigInteger.java:1599)	at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)	at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)	at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)	at javax.crypto.Cipher.doFinal(Cipher.java:2087)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)	- locked &lt;14b76357&gt; (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)	at com.google.common.cache.LocalCache.get(LocalCache.java:3970)	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)	at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)	at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXML(DefaultJiraIssuesManager.java:90)	at com.atlassian.confluence.extra.jira.CacheJiraIssuesManager.retrieveXML(CacheJiraIssuesManager.java:41)	at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXMLAsChannel(DefaultJiraIssuesManager.java:172)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.populateContextMapForStaticCountIssues(JiraIssuesMacro.java:907)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.createContextMapFromParams(JiraIssuesMacro.java:495)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.execute(JiraIssuesMacro.java:1195)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)	at java.util.concurrent.FutureTask.run(FutureTask.java:262)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)	at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:	- locked &lt;79868617&gt; (a java.util.concurrent.ThreadPoolExecutor$Worker)"Jira macros executor:thread-3" - Thread t@397   java.lang.Thread.State: RUNNABLE	at java.math.BigInteger.mulAdd(BigInteger.java:1930)	at java.math.BigInteger.montReduce(BigInteger.java:1880)	at java.math.BigInteger.oddModPow(BigInteger.java:1850)	at java.math.BigInteger.modPow(BigInteger.java:1599)	at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)	at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)	at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)	at javax.crypto.Cipher.doFinal(Cipher.java:2087)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)	- locked &lt;24bb575c&gt; (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)	at com.google.common.cache.LocalCache.get(LocalCache.java:3970)	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)	at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)	at com.atlassian.confluence.plugins.jiracharts.Base64JiraChartImageService.getBase64JiraChartImageModel(Base64JiraChartImageService.java:39)	at com.atlassian.confluence.plugins.jiracharts.render.JiraImageChart.getImageSourceModel(JiraImageChart.java:53)	at com.atlassian.confluence.plugins.jiracharts.render.PieChart.setupContext(PieChart.java:61)	at com.atlassian.confluence.plugins.jiracharts.JiraChartMacro.execute(JiraChartMacro.java:93)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)	at java.util.concurrent.FutureTask.run(FutureTask.java:262)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)	at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:	- locked &lt;5535955e&gt; (a java.util.concurrent.ThreadPoolExecutor$Worker)"Jira macros executor:thread-4" - Thread t@420   java.lang.Thread.State: RUNNABLE	at java.math.BigInteger.mulAdd(BigInteger.java:1930)	at java.math.BigInteger.squareToLen(BigInteger.java:1319)	at java.math.BigInteger.oddModPow(BigInteger.java:1849)	at java.math.BigInteger.modPow(BigInteger.java:1599)	at org.bouncycastle.crypto.engines.RSACoreEngine.processBlock(Unknown Source)	at org.bouncycastle.crypto.engines.RSABlindedEngine.processBlock(Unknown Source)	at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)	at javax.crypto.Cipher.doFinal(Cipher.java:2087)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.generateV2EncryptedCertificate(BouncyCastleEncryptionProvider.java:332)	at com.atlassian.security.auth.trustedapps.BouncyCastleEncryptionProvider.createEncryptedCertificate(BouncyCastleEncryptionProvider.java:265)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:93)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.access$000(DefaultCurrentApplication.java:16)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:32)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication$1.load(DefaultCurrentApplication.java:29)	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)	- locked &lt;51e4faa7&gt; (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)	at com.google.common.cache.LocalCache.get(LocalCache.java:3970)	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4834)	at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4839)	at com.atlassian.security.auth.trustedapps.DefaultCurrentApplication.encode(DefaultCurrentApplication.java:79)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:67)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.signRequest(TrustedRequest.java:58)	at com.atlassian.applinks.core.auth.trusted.TrustedRequest.execute(TrustedRequest.java:38)	at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXML(DefaultJiraIssuesManager.java:90)	at com.atlassian.confluence.extra.jira.CacheJiraIssuesManager.retrieveXML(CacheJiraIssuesManager.java:41)	at com.atlassian.confluence.extra.jira.DefaultJiraIssuesManager.retrieveXMLAsChannel(DefaultJiraIssuesManager.java:172)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.populateContextMapForStaticCountIssues(JiraIssuesMacro.java:907)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.createContextMapFromParams(JiraIssuesMacro.java:495)	at com.atlassian.confluence.extra.jira.JiraIssuesMacro.execute(JiraIssuesMacro.java:1195)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:78)	at com.atlassian.confluence.extra.jira.executor.StreamableMacroFutureTask.call(StreamableMacroFutureTask.java:19)	at java.util.concurrent.FutureTask.run(FutureTask.java:262)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)	at java.lang.Thread.run(Thread.java:745)   Locked ownable synchronizers:	- locked &lt;64a41f41&gt; (a java.util.concurrent.ThreadPoolExecutor$Worker)
```


Answers:
author: Dieter Greiner
body: Notice exactly the same when many JIRA issue links are on a confluence page, e.g. by JIRA issues created via JIRA issues macro


author: Chad Barnes
body: We scheduled a restart of JIRA and cleaned out all the clean-able caches. We again do not have this problem. Will keep an eye on it and update here if it recurs.


author: Chad Barnes
body: JIRA 6.3.8Confluence 5.6.3


author: Chad Barnes
body: May be related to "[https://ecosystem.atlassian.net/browse/TRUST-44"](https://ecosystem.atlassian.net/browse/TRUST-44&quot);


Tags: confluence-administration

jira

performance