Confluence is not accessible periodically
Asked by Moses Thomas:

Dear Atlassian,

I would have to ask, as i am experiencing a weird behavior, on and on we are have this problem that Confluence 7.1.0 worked for some time and then stops working periodically

```
?2020-02-10 13:56:34,794 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:56:36,833 WARN [http-nio-8011-exec-118] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:57:15,640 WARN [pool-25-thread-1] [atlassian.confluence.util.ReadOnlyLicenseCalculator] getNumberOfLicensedUsers Licensing calculation took more than PT2S
2020-02-10 13:59:50,204 ERROR [Catalina-utility-1] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Cannot get a connection, general error
2020-02-10 13:59:50,402 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@3c1bf309 took 175066 ms (warning threshold is 5000 ms)
2020-02-10 13:59:50,404 WARN [Catalina-utility-1] [internal.core.service.VCacheLock] lockWithTimeout Interrupted whilst waiting for a lock on cache:
2020-02-10 13:59:50,405 WARN [Catalina-utility-1] [confluence.impl.vcache.SynchronousExternalCache] lambda$get$11 Failed to read entry from cache 'com.atlassian.bandana.BandanaPersister': Failed due to UNCLASSIFIED_FAILURE
```

> I have performed upgrade for an existing application since December 2019 and it has been working smoothly until now. I am wondering, when i downloaded the installation file atlassian-confluence-7.1.0.tar.gz and extracted the file, according to the documentation here <https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html> in the web.xml the below element was not present in this file and we have never added this **element and currently this element is not present** and we have been using Confluence over 8yrs, the Mysql database server is self hosted.

* **Now my question is could this be the reason why i am having break in accessing Confluence within various intervals? or is this a confluence bug?** could some one please assist, this is critical issue for our production instance, and affects the quality of service.

*web.xml element*
> 4. Configure the Confluence web application
>
> Configure Confluence to use this datasource:
>
> 1. Edit `<CONFLUENCE_INSTALLATION>/confluence/WEB-INF/web.xml`.
>
> 2. Insert the following element just before `</web-app>` near the end of the file:
>
> `<resource-ref> <description>Connection Pool</description> <res-ref-name>jdbc/confluence</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth>`

* **Is the warning in logs, causing this issue** ? I doubt so, or maybe some kind of bug, currently i am not able to identify the root cause from the logs

Some time such kind of errors occurred

```
2020-02-03 13:28:10,047 WARN?[http-nio-8011-exec-111]?[confluence.util.profiling.DefaultActivityMonitor]?close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1580732802678, threadId=381, threadName='http-nio-8011-exec-111', userId='<unknown>', type='web-request', summary='/label/GPPPO/application+auto+cloud+engineering_team+extended_profile+fog+kpi+maxwell10+parameters+pre+predev+profile+redirection+sla+tool'}

```

Confluence asynchronous events(lots) are generated faster than Confluence can process them this queue is backed up

Ticket regarding this issue reported to Atlassian https://jira.atlassian.com/browse/CONFSERVER-55028

```
2020-02-10 16:32:22,979 WARN [http-nio-8011-exec-184] [atlassian.confluence.event.MonitorableCallerRunsPolicy] rejectedExecution Queue Overflow happened: Runnable: [Invoker: SingleParameterMethodListenerInvoker{method=public void com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl.onClearCache(java.lang.Object), listener=com.atlassian.pocketknife.internal.querydsl.cache.PKQCacheClearerImpl@5bbe2b61} (timed); Event: HttpRequestStatsEvent{requestCorrelationId=bab0b2eb69ad6ecb, url=label.maxwell+penguin+sw.%, key=Optional.empty, reqTime=15613, reqStartTime=1581348722407, dbReqTimesInMicros=, dbReqFinishTimes=, requestUserTime=80, requestCpuTime=84, requestGCDuration=125, requestGCCount=5, timingEventKeys=labelsDecoratorStart,labelsDecoratorFinished,sitemeshDecoratorApplyStarted,mainDecoratorStart,confluenceResourceTagsStart,confluenceResourceTagsEnd,mainDecoratorBodyStart,commonsHeaderStart,commonsHeaderFinished,mainDecoratorContentDivStart,mainDecoratorContentDivFinished,mainDecoratorFinished,sitemeshDecoratorApplyFinished,serverRenderEnd, timingEventMillis=15451,15451,15472,15474,15481,15507,15508,15508,15521,15521,15521,15523,15523,15613}]; Executor: [java.util.concurrent.ThreadPoolExecutor@24607637[Running, pool size = 4, active threads = 4, queued tasks = 384, completed tasks = 732]]
```

I tried the workaround stated here but still no success still fighting with the issue

**Confluence Bug** : Ticket <https://jira.atlassian.com/browse/CONFSERVER-40860>


Answers:
author: Moses Thomas
body: Interesting, that Confluence is vulnerable this way.

*
  > Well after deep diving, it occurs that we were attack(since we had public spaces ) and we block this user IP address in Fire wall
*
  > One of my colleague also was able to bring tomcat down with his weird JavaScript so we disable this script.
*
  > Now Confluence has been running smooth for 2days now.

I have embark on implementing a deeper monitoring for Confluence server, especially tomcat.


author: JP _AC Bielefeld Leader_
body: Hi,

we're using Confluence with a datasource connection since 5 years w/o any problems. You might want to check the max. # of connections on the MySQL side and on the datasource configuration. These numers should match or at least there should be more max. connections on the server side than defined by the data source.

Did you monitor the connection usage on the MySQL server side?

Best

JP

comments: author: Moses Thomas
body: Hi,

Thank you for your reply,

Well from mysql server side the max_connection is 151 ( for both Jira and Confluence so quite OK)

and in Confluence has max_total 60 in the data source in configuration.

**After monitoring this was the Statistics**
> Aborted_connects 126
>
> Connections = 370129
>
> Max_used_connections = 109
>
> Threads connected =29

So I don't think this is the root cause, I will keep deep digging to find the root cause

Best regards,

Moses


author: JP _AC Bielefeld Leader_
body: Hi Moses,

you mentioned in your first post that you added the Connection Pool fragment to the web.xml of Confluence.

Did you also modify the server.xml of the Tomcat server and the confluence.cfg.xml as described in

<https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html>

You might want to compare your setting with the settings we're using in our prod enviroment:

%CONFLUENCE_HOME%/conf/context.xml

```
<?xml version='1.0' encoding='utf-8'?>
<!-- The contents of this file will be loaded for each web application -->
<Context>

<!-- Default set of monitored resources. If one of these changes, the -->
<!-- web application will be reloaded. -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
<WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->

<!-- Uncomment this to enable Comet connection tacking (provides events
on session expiration as well as webapp lifecycle) -->
<!--
<Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
-->

<ResourceLink type="javax.sql.DataSource"
name="jdbc/confluence"
global="jdbc/confluence" />

<ResourceLink type="javax.mail.Session"
name="mail/mailsession"
global="mail/mailsession" />

</Context>
```

%CONFLUENCE_HOME%/conf/server.xml

```
<Server port="8000" shutdown="SECRET">
<!-- Allow JMX access thru local firewall using defined ports -->
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="SECRET" rmiServerPortPlatform="SECRET" />

<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>

<Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
username="SECRET"
password="SECRET"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://SECRET/confluence?sessionVariables=storage_engine%3DInnoDB&amp;useUnicode=true&amp;characterEncoding=utf8" 
defaultTransactionIsolation="READ_COMMITTED"
validationQuery="SELECT 1"
initialSize="20"
maxTotal="400"
maxIdle="20"
minIdle="0" 
maxWaitMillis="-1"
timeBetweenEvictionRunsMillis="30000"
minEvictableIdleTimeMillis="60000"
testOnBorrow="true"
removeAbandonedOnBorrow="false"
removeAbandonedOnMaintenance="false"
removeAbandonedTimeout="120"
logAbandoned="true"
/>

</GlobalNamingResources>

<Service name="Tomcat-Standalone">

<Connector port="8009"
minSpareThreads="10"
maxThreads="150"
address="SECRET" 
protocol="AJP/1.3"
redirectPort="8444" 
enableLookups="false"
URIEncoding="UTF-8"
maxPostSize="-1"
packetSize="65536"/>

<Engine name="Standalone" defaultHost="SECRET" jvmRoute="node0">

<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>

<Host name="SECRET" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4">

<Context path="/confluence" docBase="../confluence" reloadable="false" useHttpOnly="true">
<Manager pathname="" />

<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context>

<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0" reloadable="false" useHttpOnly="true">
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60" />
</Context> 
</Host>

</Engine>

</Service>
</Server>
```

%CONFLUENCE_DATA%/confluence.cfg.xml

```
<?xml version="1.0" encoding="UTF-8"?>

<confluence-configuration>
<setupStep>complete</setupStep>
<setupType>custom</setupType>
<buildNumber>8100</buildNumber>
<properties>
...
<property name="hibernate.connection.datasource">java:comp/env/jdbc/confluence</property>
<property name="hibernate.database.lower_non_ascii_supported">true</property>
<property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.MySQLDialect</property>
<property name="hibernate.setup">true</property>
...
</properties>
</confluence-configuration>
```

This configuration makes use of the Tomcat Database Pool handling, which is much easier to monitor by JMX. If you only changed the web.xml, you will run into your mentioned problems.

Could you post the relevant section of your server.xml?

Best

JP


author: JP _AC Bielefeld Leader_
body: Oh, I forgot to add the modification to the %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml right before the end of the file:

```
...

 <resource-ref>
<description>Connection Pool</description>
<res-ref-name>jdbc/confluence</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>

</web-app>
```


author: Moses Thomas
body: [@JP _AC Bielefeld Leader_](/t5/user/viewprofilepage/user-id/1135065) Thanks, I have gone through your post, now suppose all config files are Okay, and **I DONT** **HAVE** the below element **what do i stand to face** ? cos according to my previous explanation Confluence has been working without the below element in %CONFLUENCE_HOME%/confluence/WEB_INF/web.xml until lately so i wounder if this could be the cause? I doubt so.

> ```
> <resource-ref>
> <description>Connection Pool</description>
> <res-ref-name>jdbc/confluence</res-ref-name>
> <res-type>javax.sql.DataSource</res-type>
> <res-auth>Container</res-auth>
> <res-sharing-scope>Shareable</res-sharing-scope>
> </resource-ref>
>
> </web-app>
> ```

Kind regards,

Mo


author: JP _AC Bielefeld Leader_
body: Hi,

database connections in a Tomcat application server environment can be handled by the application or the Tomcat server itself as a shareable resource. Configuring a shareable resource has the advantage for easier monitoring (JMX) and having a central configuration file (server.xml). Imaging having several applications on a Tomcat server \& the DB server changes. You have to modify each application. With a shared resource you just need to change one file.

The fragment in the web.xml tells Confluence to use a shared SQL Datasource named jdbc/confluence defined in the container configuration (server.xml).

Maybe you haven't been using a datasource connection before, but just a also working application JDBC connection.

Great you solved it.

Best

JP


Tags: bug

confluence-server

performance