Bitbucket Cluster - Node Authentication Failed
Asked by Thomas Langhorne:

Having trouble getting clustering working for bitbucket.  

Running with Bigbang, istio is the service mesh, and matallb is the load balancer  

<br />

Helm Chart Version 1.16.6  
Bitbucket Application Version: 8.5.0  
Helm chart Values:  

```
values:
  image:
    repository: docker.io/atlassian/bitbucket
    tag: "8.5.0"
  serviceAccount:
    clusterRole:
      create: true
    clusterRoleBinding:
      create: true
  database:
    type: "postgresql"
    url: "jdbc:postgresql://modbucket-postgres-postgresql:5432/bitbucket"
    driver: "org.postgresql.Driver"
    credentials:
      secretName: atlassian-postgresdb
      usernameSecretKey: postgres-user
      passwordSecretKey: postgresql-password
  bitbucket:
    clustering:
      enabled: true
    hazelcastService:
      enabled: true
    additionalEnvironmentVariables:
      - name: "http_proxy"
        value: "REDACTED"
      - name: "https_proxy"
        value: "REDACTED"
      - name: "no_proxy"
        value: "REDACTED"
    additionalJvmArgs:
      - -Dcom.redhat.fips=false
      - -Djava.net.preferIPv4Stack=true
      - -Datlassian.dev.mode=false
      - -Datlassian.plugins.enable.wait=300
  ingress:
    nginx: false
    # Disable SSL Configuration as Istio Ingress Controller will provide SSL configuration
    # If this is "true" then default Bitbucket security configuraitons will prevent web page resolution
    https: false
  monitoring:
    exposeJmxMetrics: true
    jmxExporterImageRepo: docker.io/bitnami/jmx-exporter
  podAnnotations:
    traffic.sidecar.istio.io/excludeOutboundPorts: "80,443"
  podLabels:
    app: modbucket
  volumes:
    localHome:
      persistentVolumeClaim:
        create: true
        resources:
          requests:
            storage: 10Gi
      mountPath: "/var/atlassian/application-data/bitbucket"
    sharedHome:
      nfsPermissionFixer:
        imageRepo: docker.io/library/alpine
        imageTag: 3.18.2
      persistentVolumeClaim:
        storageClassName: ceph-filesystem
        create: true
        resources:
          requests:
            storage: 10Gi
      mountPath: "/var/atlassian/application-data/shared-home"
```

<br />

<br />

The issue I am experiencing is that the Scaled Nodes:  
1) The additional nodes don't show up under the webpage/admin/clustering  
2) logs have the error **Node authentication failed: Unexpected bytes from remote node, closing socket
<br />**I will post the logs as a reply


Answers:
author: Thomas Langhorne
body: Posting the logs for additional details  

Log from my 2nd pod, that fails Node authentication  

INFO:root:User is currently root. Will downgrade run user to bitbucket  
INFO:root:Running Bitbucket with command '/opt/atlassian/bitbucket/bin/exec-bitbucket-node.sh', arguments \['/opt/atlassian/bitbucket/bin/exec-bitbucket-node.sh'\]  
Starting Atlassian Bitbucket as the current user  
Starting Atlassian Bitbucket as the current user  
/opt/atlassian/bitbucket/bin/_exec-webapp.sh: line 62: \\nStarting Bitbucket webapp at <http://localhost:${bitbucket.http.port}${bitbucket.context>}: bad substitution  
2023-11-02 16:30:33,819 INFO \[main\] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v8.5.0 using Java 11.0.20.1 on modbucket-bitbucket-1 with PID 7 (/opt/atlassian/bitbucket/app/WEB-INF/classes started by bitbucket in /var/atlassian/application-data/bitbucket)  
2023-11-02 16:30:33,828 INFO \[main\] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to 1 default profile: "default"  
2023-11-02 16:30:34,636 INFO \[main\] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 8.5.0 (d5f13a2 built on Mon Oct 10 02:01:27 UTC 2022)  
2023-11-02 16:30:34,637 INFO \[main\] c.a.b.i.boot.log.BuildInfoLogger JVM: Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.20.1+1  
2023-11-02 16:30:35,927 INFO \[main\] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 3.147 seconds (JVM running for 4.003)  
2023-11-02 16:30:38,807 INFO \[spring-startup\] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket  
2023-11-02 16:30:42,612 INFO \[spring-startup\] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/shared-home  
WARNING: An illegal reflective access operation has occurred  
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/opt/atlassian/bitbucket/app/WEB-INF/lib/hazelcast-3.12.12.jar) to field sun.nio.ch.SelectorImpl.selectedKeys  
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer  
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations  
WARNING: All illegal access operations will be denied in a future release  
2023-11-02 16:30:44,766 WARN \[spring-startup\] c.h.s.d.integration.DiscoveryService \[10.42.1.101\]:5701 \[modbucket-bitbucket\] \[3.12.12\] Cannot fetch the current zone, ZONE_AWARE feature is disabled  
2023-11-02 16:30:44,814 WARN \[spring-startup\] c.h.kubernetes.KubernetesClient Cannot fetch public IPs of Hazelcast Member PODs, you won't be able to use Hazelcast Smart Client from outside of the Kubernetes network  
2023-11-02 16:30:44,840 WARN \[hz.hazelcast.cached.thread-3\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:60645 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:31:08,683 INFO \[spring-startup\] c.a.s.i.s.g.m.DefaultSidecarManager Sidecar started after 8281ms  
2023-11-02 16:31:13,103 INFO \[spring-startup\] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:43777  
2023-11-02 16:31:16,855 INFO \[hz.hazelcast.event-1\] c.a.s.i.s.g.m.DefaultMeshSidebandRegistry Sidecar (<http://localhost:7777>): Opening sideband channel  
2023-11-02 16:31:18,541 WARN \[spring-startup\] c.a.p.o.c.felix.ExportsBuilder Cannot deep scan non file 'jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/rome-1.18.0.jar!/rome-utils-1.18.0.jar'  
2023-11-02 16:31:18,588 WARN \[spring-startup\] c.a.p.o.c.felix.ExportsBuilder Cannot deep scan non file 'jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/atlassian-tenancy-compatibility-plugin-4.0.0.jar!/atlassian-tenancy-api-4.0.0.jar'  
2023-11-02 16:31:20,599 WARN \[FelixStartLevel\] o.e.g.b.e.i.s.ExtenderConfiguration Gemini Blueprint extensions bundle not present, annotation processing disabled.  
2023-11-02 16:31:29,488 WARN \[ThreadPoolAsyncTaskExecutor::Thread 13\] c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.admin.criteria.WritableDirectoryExistsCriteria' for component importation purposes. Skipping...  
2023-11-02 16:31:29,488 WARN \[ThreadPoolAsyncTaskExecutor::Thread 13\] c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.setup.settings.SettingsManager' for component importation purposes. Skipping...  
2023-11-02 16:31:29,488 WARN \[ThreadPoolAsyncTaskExecutor::Thread 13\] c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.user.SignupManager' for component importation purposes. Skipping...  
2023-11-02 16:31:29,488 WARN \[ThreadPoolAsyncTaskExecutor::Thread 13\] c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.util.UserChecker' for component importation purposes. Skipping...  
2023-11-02 16:31:42,256 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle com.atlassian.bitbucket.server.bitbucket-frontend; it does not exist  
2023-11-02 16:31:42,898 WARN \[spring-startup\] c.a.p.c.r.OsgiBundledPathScanner Cannot scan directory /page/ in bundle com.atlassian.bitbucket.server.bitbucket-policies as it does not exist  
2023-11-02 16:31:42,923 WARN \[spring-startup\] c.a.p.c.r.StaticPluginPathScanner Cannot scan directory /static/bitbucket/internal/feature/admin/ratelimit/ in web static as it does not exist  
2023-11-02 16:31:44,202 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist  
2023-11-02 16:31:44,481 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.de_DE; it does not exist  
2023-11-02 16:31:44,483 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist  
2023-11-02 16:31:44,486 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.fr_FR; it does not exist  
2023-11-02 16:31:44,488 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist  
2023-11-02 16:31:44,493 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.ja_JP; it does not exist  
2023-11-02 16:31:44,495 WARN \[spring-startup\] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist  
2023-11-02 16:31:44,627 INFO \[spring-startup\] c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/bin/git version 2.37.7  
2023-11-02 16:31:45,082 INFO \[spring-startup\] c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...  
2023-11-02 16:31:45,190 INFO \[spring-startup\] c.a.b.internal.ssh.server.SshServer Started SSH server successfully.  
2023-11-02 16:31:45,559 INFO \[spring-startup\] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck  
2023-11-02 16:31:45,559 INFO \[spring-startup\] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob  
2023-11-02 16:31:45,677 INFO \[spring-startup\] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: VersionCheck  
2023-11-02 16:31:45,681 INFO \[spring-startup\] c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started  
2023-11-02 16:31:45,682 INFO \[spring-startup\] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck  
2023-11-02 16:31:45,682 INFO \[spring-startup\] c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for SearchSynchronizeJob  
2023-11-02 16:31:45,682 INFO \[spring-startup\] c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck  
2023-11-02 16:32:00,627 INFO \[Caesium-1-4\] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search  
2023-11-02 16:32:00,652 WARN \[Caesium-1-4\] c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to the search server during index synchronisation.  
2023-11-02 16:32:00,653 WARN \[Caesium-1-4\] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: b259e598-83f0-471c-86b0-b88f5b6c4d80  
2023-11-02 16:32:32,406 WARN \[pool-6-thread-1\] c.a.u.p.MpacApplicationCacheManager Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: org.apache.http.conn.ConnectTimeoutException: Connect to marketplace.atlassian.com:443 \[marketplace.atlassian.com/104.192.142.15, marketplace.atlassian.com/104.192.142.16, marketplace.atlassian.com/104.192.142.17\] failed: connect timed out  
2023-11-02 16:33:17,458 WARN \[pool-6-thread-1\] c.a.u.p.MpacApplicationCacheManager Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: org.apache.http.conn.ConnectTimeoutException: Connect to marketplace.atlassian.com:443 \[marketplace.atlassian.com/104.192.142.16, marketplace.atlassian.com/104.192.142.17, marketplace.atlassian.com/104.192.142.15\] failed: connect timed out  
2023-11-02 16:35:44,508 WARN \[hz.hazelcast.cached.thread-5\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:50155 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:37:44,496 WARN \[hz.hazelcast.cached.thread-2\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:58749 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:39:44,498 WARN \[hz.hazelcast.cached.thread-2\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:49205 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:41:44,496 WARN \[hz.hazelcast.cached.thread-4\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:36281 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:43:44,492 WARN \[hz.hazelcast.cached.thread-2\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:53859 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:45:44,492 WARN \[hz.hazelcast.cached.thread-4\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:49217 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:47:44,492 WARN \[hz.hazelcast.cached.thread-5\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:37833 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket  
2023-11-02 16:49:44,495 WARN \[hz.hazelcast.cached.thread-4\] c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:58063 -\> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket


Tags: bitbucket-server

server