Remote OpenSearch connection-result CONNECTION_ERROR
Asked by Lasse Meyer:

I have a Bitbucket Data Center on a Linux VM in Azure and a remote OpenSearch on a Linux VM in Azure, both are in the same network.  
OpenSearch is configured like in: [Debian - OpenSearch Documentation](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/debian/)  
And Bitbucket Data Center is configured like in: [Install and configure a remote OpenSearch server \| Bitbucket Data Center 8.18 \| Atlassian Documentation](https://confluence.atlassian.com/bitbucketserver/install-and-configure-a-remote-opensearch-server-1115664379.html)  

When i?m using this on my bitbucket data center vm: "curl -X GET <https://IP:9200> -u 'user:pw' --insecure" (with my own credentials, which i setuped), i get the right answer from the opensearch server.  
But in Bitbucket i still have connection-result: CONNECTION_ERROR  

This is the only thing i can find in the atlassian-bitbucket.log and i doesnt get it:

*** ** * ** ***

2024-02-21 14:21:05,499 DEBUG \[I/O dispatcher 15\] c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:  
java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target  

*** ** * ** ***

<br />

And i dont know why.  
Does anyone know or have some ideas?


Answers:
author: Aman Shrivastava
body: Hi Lasse,

The below error indicates that a target application(in this case OpenSearch) is running with an SSL/TLS configuration but it can't be trusted by Bitbucket. This is most likely due to the peer application's certificate being self-issued (i.e., not signed by a CA) or a certificate chain does not exist within the Java trust store that Bitbucket is using.

```
java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
```

<br />

I would suggest you review the [Unable to connect to SSL services due to "PKIX Path Building Failed" error](https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html) KB that covers all the possible causes and the fix for this error. In case of any problem, please raise a ticket with [Atlassian support.](https://support.atlassian.com/)

Regards,  
Aman

comments: author: Lasse Meyer
body: Im sorry but this doesnt help me in any way.  
Im using self signed certs like the Bitbucket Data Center documentation tells me to do.  
But Bitbucket doesnt accept them because their are self signed?  

I dont get it.


author: Aman Shrivastava
body: Hi Lasse,  

I will try to clarify all the aspects.

* The document [Install and configure a remote OpenSearch server](https://confluence.atlassian.com/bitbucketserver/install-and-configure-a-remote-opensearch-server-1115664379.html) covers the minimum configuration required to configure the OpenSearch security plugin and in the context of that, it is mentioned that a self-signed certificate can be used if plugins.security.ssl.transport.enforce_hostname_verification is set to false.
* It's Java that gives the error when it detects the server's(OpenSearch) certificate can't be trusted. Java maintains a trust store that contains the root and intermediate certificates of public certificate authorities. Based on these certificates, java establishes trust with the server.
* If you want to use the self-signed certificate then you will need to import the complete certificate chain into the Java trust store which Bitbucket is using. The process to import the certificate in the trust store is included in the KB [Unable to connect to SSL services due to "PKIX Path Building Failed" error](https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html)  

Regards,  
Aman


author: Lasse Meyer
body: 1. Check to see that the correct truststore is in use. If `-Djavax.net.ssl.trustStore` has been configured, it will override the location of the default truststore, which will need to be checked.  

   How to do that for bitbucket? I dont find any documenations about it.  
   Because, it?s funny it doesnt fail when i check it with SSLPoke..  

   *** ** * ** ***

   root@bitbucketVM2:/bitbucket/jre/bin# ./java -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -Djavax.net.debug=ssl SSLPoke 10.28.11.16 9200  
   javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:00:26.187 UTC\|null:-1\|jdk.tls.keyLimits: entry = AES/GCM/NoPadding KeyUpdate 2\^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472  
   javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:00:26.207 UTC\|null:-1\|jdk.tls.keyLimits: entry = ChaCha20-Poly1305 KeyUpdate 2\^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472  
   javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:00:26.484 UTC\|null:-1\|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
   countdown value = 137438953472  
   javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:00:26.486 UTC\|null:-1\|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
   countdown value = 137438953472  
   javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:00:26.543 UTC\|null:-1\|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
   countdown value = 137438953472  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.546 UTC\|null:-1\|No X.509 cert selected for EC  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.546 UTC\|null:-1\|No X.509 cert selected for EC  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.546 UTC\|null:-1\|No X.509 cert selected for EC  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.547 UTC\|null:-1\|No X.509 cert selected for EdDSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.548 UTC\|null:-1\|No X.509 cert selected for EdDSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.549 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.550 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.550 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.550 UTC\|null:-1\|No X.509 cert selected for RSASSA-PSS  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.551 UTC\|null:-1\|No X.509 cert selected for RSASSA-PSS  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.552 UTC\|null:-1\|No X.509 cert selected for RSASSA-PSS  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.553 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.553 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.554 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.556 UTC\|null:-1\|No X.509 cert selected for EC  
   javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:00:26.557 UTC\|null:-1\|No X.509 cert selected for RSA  
   javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:00:26.565 UTC\|null:-1\|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
   countdown value = 137438953472  
   Successfully connected  


author: Lasse Meyer
body: root@bitbucketVM2:/bitbucket/jre/bin# ./java -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -Djavax.net.debug=ssl SSLPoke 10.28.11.16 9200  
javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:06:00.023 UTC\|null:-1\|jdk.tls.keyLimits: entry = AES/GCM/NoPadding KeyUpdate 2\^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472  
javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:06:00.071 UTC\|null:-1\|jdk.tls.keyLimits: entry = ChaCha20-Poly1305 KeyUpdate 2\^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472  
javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:06:00.640 UTC\|null:-1\|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
countdown value = 137438953472  
javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:06:00.644 UTC\|null:-1\|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
countdown value = 137438953472  
javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:06:00.701 UTC\|null:-1\|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
countdown value = 137438953472  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.703 UTC\|null:-1\|No X.509 cert selected for EC  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.713 UTC\|null:-1\|No X.509 cert selected for EC  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.714 UTC\|null:-1\|No X.509 cert selected for EC  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.714 UTC\|null:-1\|No X.509 cert selected for EdDSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.715 UTC\|null:-1\|No X.509 cert selected for EdDSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.715 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.715 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.716 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.716 UTC\|null:-1\|No X.509 cert selected for RSASSA-PSS  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.717 UTC\|null:-1\|No X.509 cert selected for RSASSA-PSS  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.717 UTC\|null:-1\|No X.509 cert selected for RSASSA-PSS  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.718 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.718 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.718 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.719 UTC\|null:-1\|No X.509 cert selected for EC  
javax.net.ssl\|ALL\|10\|main\|2024-02-23 09:06:00.719 UTC\|null:-1\|No X.509 cert selected for RSA  
javax.net.ssl\|DEBUG\|10\|main\|2024-02-23 09:06:00.722 UTC\|null:-1\|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE  
countdown value = 137438953472  
Successfully connected


author: Lasse Meyer
body: Im not sure what step helped me to get further, but this is my problem now:  
Why does it say it doesnt match the CN, when in CN the correct IP is?

*** ** * ** ***

2024-02-23 09:35:42,596 DEBUG \[I/O dispatcher 5\] c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:  
java.util.concurrent.CompletionException: javax.net.ssl.SSLPeerUnverifiedException: Host name '10.28.11.16' does not match the certificate subject provided by the peer (CN=10.28.11.16, OU=UNIT, O=ORG, L=TORONTO, ST=ONTARIO, C=CA)  
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)  
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)  
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)  
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)  
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)  
at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor$1.failed(ApacheRequestExecutor.java:160)  
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)  
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)  
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:432)  
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)  
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:82)  
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:40)  
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:156)  
at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:187)  
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)  
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)  
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)  
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)  
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)  
at java.base/java.lang.Thread.run(Unknown Source)  
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name '10.28.11.16' does not match the certificate subject provided by the peer (CN=10.28.11.16, OU=UNIT, O=ORG, L=TORONTO, ST=ONTARIO, C=CA)  
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.verifySession(SSLIOSessionStrategy.java:217)  
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy$1.verify(SSLIOSessionStrategy.java:197)  
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:372)  
at org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:588)  
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:154)  
... 7 common frames omitted


Tags: bitbucket-data-center

bitbucket-server

data-center

server