Search - Connection attempt failed to establish connection
Asked by Eric Zimmerman:

**Environment**

We run a single Bitbucket Data Center server on Ubuntu 22.04 on airgap networks.

We upgraded all of the instances to v8.9.2 about a month ago and all seemed to be well.

**Issue**

Within the past few days, one of them is exhibiting problem with search.

When I press the "Server settings" "Test" button, the 'atlassian-bitbucket.log', shows:

```
# note this is an except since I'm manually typing from an airgap server

DEBUG [pool-53-thread-1]  c.a.b.i.s.c.s.t.DefaultSearchConnectionTester Testing connection with search server failed due to exception:
java.util.concurrent.CompletionException: java.net.ConnectException: Connection refused

...

caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)

```

Notes

* stopped and restarted a number of times
* rebooted the server a number of times
* it seems like search is running on the proper ports
* I ran through a number of things noted here: <https://confluence.atlassian.com/bitbucketserverkb/troubleshooting-steps-for-bitbucket-server-code-search-827125334.html>
* Why else might connection be refused?


Answers:
author: Mahesh Shinde
body: Hi [@Eric Zimmerman](/t5/user/viewprofilepage/user-id/789410) :

Have you configured the remote search server or is it bundle with bitbucket application?

Please, validate the below things on your server.

* `tail -f $BITBUCKET_HOME/log/search/bitbucket_search.log`
  * Nothing happens
* `ps -ef | grep `search
  * reveals search server is up and running
* `netstat -tulnp`
  * as `atlbitbucket` user does not show the port 7992 on listen. You can't see the PID of your search server listened
* `telnet localhost 7992`
  * Connection is refused

This will give some clarity where is the exact issue.

Regards,

Mahesh Shinde

comments: author: Eric Zimmerman
body: *re: \`tail -f $BITBUCKET_HOME/log/search/bitbucket_search.log\`*

I see log entries from todays date, ending with \`\[bitbucket_bundled\] Running full sweep\`

*re: \`ps -ef \| grep search\`*

It shows the command line for when it was started yesterday (uses /opt/atlassian/bitbucket/8.9.2/jre/bin/java and opensearch paths look fine.

*re: \`netstat -tulnp\`*

I don't have netstat, but the I ran the following \`sudo -u atlbitbucket ss -tulnp\`.

Ports 7992 and 7993 are displayed as listening with uses "java" and listed pids.

*re: \`telnet localhost 7992\`*

\`Connected to localhost.\`

One other note is that we moved from Windows to Linux last month and the other equivalent server does not exhibit this problem.


author: Mahesh Shinde
body: Hi [@Eric Zimmerman](/t5/user/viewprofilepage/user-id/789410) :

As you are using inbuild search server, please access the below URL on browser and provide the credentials to login.   

<http://localhost:7992/>

If you **don't have a local browser** on the server that is running Bitbucket Server, you should also be able to run the following command *successfully* :

`$ curl -u username:password `<http://localhost:7992>` `


author: Mahesh Shinde
body: If additional troubleshooting is required and the debug log level of the Bitbucket Server to Search server is required, the following command will enable it:

`curl -u <ADMIN_USERNAME> -v -X PUT -d "" -H "Content-Type: application/json" <BASE_URL>/rest/api/latest/logs/logger/com.atlassian.bitbucket.search.internal.indexing/debug`


author: Eric Zimmerman
body: Thankyou for the feedback.

I found some notes from someone who encountered this before.
> Check the configuration file of opensearch. The file name is "opensearch.yml" and it can be found in "\<BITBUCKET_HOME\>/config/search".
>
> The IP address is determined by "network.host". By default, it may be set to "_local_". If the 7992 port was shown in the netstat results, but the search server test failed, try changing the setting to "network.host = \[_local_, _site_\]".
>
> Check that the HTTP TCP port is set to 7992.

This ended up solving the issue after I changed this and restarted Bitbucket.

So, my question is why? ... our other servers do ++**not**++have this set, but search works just fine, so ???


Tags: bitbucket-server

server