Database pool Connection is not available
Asked by Graham Twine:

Hello Confluence admins,

I have recently upgraded Confluence from 7.19.17 to 8.5.12.  

This is a two node cluster and has been working just fine until the upgrade.

The cluster is connected to an SQL server and uses kerberos to authenticate.

Below the configuration used and modified to remove sensitive info.

The pool size has been 60 for the last few years and I have increased it to 80 in an attempt to stabilize the service

Configuration {#toc-hId--349739952}
-----------------------------------

#### confluence.cfg.xml {#toc-hId--1456129837}

```
   <property name="hibernate.c3p0.max_size">80</property> 
   <property name="hibernate.c3p0.validate">true</property> 
   <property name="hibernate.connection.autocommit">false</property> 
   <property name="hibernate.connection.driver_class">com.microsoft.sqlserver.jdbc.SQLServerDriver</property> 
   <property name="hibernate.connection.isolation">2</property> 
   <property name="hibernate.connection.password">REDACTED</property> 
   <property name="hibernate.connection.provider_class">com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider</property> 
   <property name="hibernate.connection.url">jdbc:sqlserver://HOST:PORT;databaseName=DATABASE;domain=DOMAINNAME;integratedSecurity=true;authenticationScheme=JavaKerberos</property> 
   <property name="hibernate.connection.username">REDACTED</property> 
   <property name="hibernate.database.lower_non_ascii_supported">true</property> 
   <property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.SQLServerDialect</property> 
   <property name="hibernate.hikari.idleTimeout">30000</property> 
   <property name="hibernate.hikari.maximumPoolSize">80</property> 
   <property name="hibernate.hikari.minimumIdle">20</property> 
   <property name="hibernate.hikari.registerMbeans">true</property>
```

#### server.xml {#toc-hId-1031382996}

```
        <Connector port="8080" connectionTimeout="20000" redirectPort="8443" 
                   maxThreads="48" maxPostSize="16777216" minSpareThreads="10" 
                   maxHttpHeaderSize="24578" enableLookups="false" acceptCount="10" URIEncoding="UTF-8" 
                   protocol="org.apache.coyote.http11.Http11NioProtocol" 
                   scheme="https" secure="true" proxyName="confluence.domain.tld" proxyPort="443" />
 
```

Issue {#toc-hId--1477136045}
----------------------------

#### Error {#toc-hId-1711441366}

At least twice a day one of the two nodes will stop responding. Sometimes they both stop responding. The logs fill up with 'Unable to aquire JDBC Connection' due to no connections in the pool, due to No loginModule found. (See Exception Exanmple below)  

#### Analysis {#toc-hId--96013097}

The DBA team are confident there are no issues and they do not see failed authentication attempts.

Restarting Confluence brings the node online and it starts accepting traffic, logs indicate normal operation

#### Observation {#toc-hId--1903467560}

The bundled SQL Server driver is used and quite old (7.4) [mssql-jdbc-7.4.1.jre8.jar](https://learn.microsoft.com/en-us/sql/connect/jdbc/release-notes-for-the-jdbc-driver?view=sql-server-2017#-741)

From the [database-jdbc-drivers](https://confluence.atlassian.com/conf85/database-jdbc-drivers-1283360987.html) documentation

We recommend that you ++use the bundled++ **Type 4** JDBC ++driver++.

If you decide to use a later version, we may not be able to provide support for any problems you encounter.

The bundled jar is only for java 8 if it does hwat it says on the tin

Java {#toc-hId--117019305}
--------------------------

I do not find a bundled version of Java in the gunzip archive

|           Confluence version           |            Supported Java vendors            | Bundled Java vendor |
|----------------------------------------|----------------------------------------------|---------------------|
| 7.13.2 to 7.13.x, and 7.14.1 to latest | Oracle JDK/JRE Adopt OpenJDK Eclipse Temurin | Eclipse Temurin     |

++**EDIT:**++   
From [supported-platforms](https://confluence.atlassian.com/conf85/supported-platforms-1283361429.html) the JRE is not expected to be in the gunzip file. It is only in the installer

**Good to know:**

* You don't need to install Java++if you plan to use the installer++ to install Confluence, as a Temurin Java 17 JRE is bundled with Confluence.

**END EDIT**

Exception Example {#toc-hId--1924473768}
----------------------------------------

```
2024-08-12 01:31:30,166 WARN [Caesium-1-4] [confluence.impl.search.IndexFlushScheduler] lambda$new$1 Failed to flush index queue LuceneContentIndexFlusher, retry in 10s 
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Co 
nnection 
       at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:600) 
       at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:31) 
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) 
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) 
       at jdk.internal.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) 
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
       at java.base/java.lang.reflect.Method.invoke(Method.java:566) 
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) 
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234) 
       at com.sun.proxy.$Proxy107.getTransaction(Unknown Source) 
       at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) 
       at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) 
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) 
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) 
       at com.sun.proxy.$Proxy239.flushQueue(Unknown Source) 
       at com.atlassian.confluence.impl.search.IndexFlushScheduler.lambda$new$1(IndexFlushScheduler.java:122) 
       at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:121) 
       at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:83) 
       at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134) 
       at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106) 
       at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90) 
       at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:464) 
       at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:431) 
       at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:409) 
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66) 
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60) 
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35) 
       at java.base/java.lang.Thread.run(Thread.java:829) 
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection 
       at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48) 
       at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37) 
       at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) 
       at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) 
       at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111) 
       at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138) 
       at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:516) 
       at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:502)
        ... 28 more 
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms. 
       at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) 
       at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) 
       at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) 
       at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) 
       at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.getConnection(HikariCPConnectionProvider.java:77) 
       at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.getConnection(DelegatingHikariConnectionProvider.java:48) 
       at jdk.internal.reflect.GeneratedMethodAccessor237.invoke(Unknown Source) 
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
       at java.base/java.lang.reflect.Method.invoke(Method.java:566) 
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) 
       at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) 
       at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:113) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) 
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) 
       at com.sun.proxy.$Proxy743.getConnection(Unknown Source) 
       at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38) 
       at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108) 
       ... 31 more



Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Kerberos Login failed: Integrated authentication failed. ClientConnectionId:bc429792-b23f-4c23-9af8-b98f1a769208 due to javax.security.auth.login.Logi
nException (No LoginModule found for com.sun.security.auth.module.Krb5LoginModule) 
       at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:100) 
       at com.microsoft.sqlserver.jdbc.KerbAuthentication.generateClientContext(KerbAuthentication.java:207) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4820) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3731) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$000(SQLServerConnection.java:94) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3675) 
       at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2979) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2488) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2142) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1993) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1164) 
       at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:760) 
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) 
       at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) 
       at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) 
       at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) 
       at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71) 
       at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726) 
       at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) 
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
       ... 1 more 
Caused by: javax.security.auth.login.LoginException: No LoginModule found for com.sun.security.auth.module.Krb5LoginModule 
       at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:731) 
       at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672) 
       at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670) 
       at java.base/java.security.AccessController.doPrivileged(Native Method) 
       at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670) 
       at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:581) 
       at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:77) 
       ... 23 more

```


Answers:
author: Graham Twine
body: The ms-sql driver shipped is now EOL and it is also for Java 8.  
The error is due to the way Java 8 and Java 11 use ForkJoinPool.

:( If you are using MS SQL then Confluence is not really compatible with Java 11.


Tags: confluence-data-center

data-center

jdbc