under Kubernetes env confluence cluster node won't join
Asked by Alexander Tchaikovski:

Hello Colleagues. I've been trying to make a cluster out of K8S confluence Datacenter app. one node is ok, 2nd one won't start with strange error:  
2022-05-20 12:39:57,932 ERROR \[hz.confluence.priority-generic-operation.thread-0\] \[com.hazelcast.cluster\] log \[10.0.129.104\]:5701 \[wiki-confluence\] \[3.12.11\] Node could not join cluster. A Configuration mismatch was detected: Incompatible joiners! expected: multicast, found: tcp-ip Node is going to shutdown now!  
2022-05-20 12:39:57,932 WARN \[hz.confluence.priority-generic-operation.thread-0\] \[com.hazelcast.instance.Node\] log \[10.0.129.104\]:5701 \[wiki-confluence\] \[3.12.11\] Terminating forcefully...  
2022-05-20 12:40:01,936 ERROR \[Catalina-utility-1\] \[com.hazelcast.instance.Node\] log \[10.0.129.104\]:5701 \[wiki-confluence\] \[3.12.11\] Could not join cluster. Shutting down now!  
2022-05-20 12:40:02,006 WARN \[Catalina-utility-1\] \[com.hazelcast.util.PhoneHome\] log \[10.0.129.104\]:5701 \[wiki-confluence\] \[3.12.11\] Could not schedule phone home task! Most probably Hazelcast failed to start.  
2022-05-20 12:40:02,028 ERROR \[Catalina-utility-1\] \[atlassian.confluence.setup.ConfluenceConfigurationListener\] contextInitialized An error was encountered while bootstrapping Confluence (see below):  
Node failed to start!  
I believe there's something I missed to config but can't find what it is. Shared home is in place, data is ok, configured with multicast and didn't change anything else.


Answers:
author: Alexander Tchaikovski
body: ok, got it. but in case it's a permission thing where can i read about the permissions? i suspect that shared-home has a wrong permissions. what the should be? because helm chart don't mention those, internet won't give much info on that. i mean at the moment root:root has all rights, they're drwxrwxrwx and i'm concerned. is it correct? if it's not how to set them up correctly? thank you.


author: Yevhen
body: [@Alexander Tchaikovski](/t5/user/viewprofilepage/user-id/4897469) I am afraid I can't suggest anything new - it's either disc space and/or permissions. Doesn't look like a k8s specific issue - just a [general issue](https://confluence.atlassian.com/confkb/failed-to-propagate-index-in-confluence-data-center-7-7-and-later-1005783333.html#:~:text=Make%20sure%20the%20user%20you,and%20each%20local%20home%20directory.).


author: Alexander Tchaikovski
body: might be. we got it fixed, it's ok.

there's another thing though. we've got some pics/thumbnails missing so tried to make reindex and propagate. but it throws an error: **Failed to propagate.** Can't access shared home... checked, shared-home is available on both nodes, users can read/write, free space is not an issue... tried to shut down 1 node, reindex, then up 2nd node and do it again - same error. is there any debug option or specific log file so we can use them? thank you!


author: Yevhen
body: Perhaps the first time you migrated cluster properties which don't work in Kube. It's just my assumption as I don't have all the data to leap to a data-proven conclusion.


author: Alexander Tchaikovski
body: Hello [@Yevhen](/t5/user/viewprofilepage/user-id/3965218) . thank you for your comment.

I gave up finally, installed STS latest version via Helm from scratch and migrated data using UI (backup/restore).

With these options SW installs smoothly, forms a cluster and works fine. Our luck we don't have much data so backup/restore is working ok, thinking of a case when we need to migrate 100Gb or more..


author: Yevhen
body: [@Alexander Tchaikovski](/t5/user/viewprofilepage/user-id/4897469) it looks like K8s Hazelcast discovery does not happen. Is clustering turned on in values.yaml? Also, is this a new installation or you migrated an existing Confluence to Kube? Perhaps, there are leftovers of properties from a previous installation?


Tags: confluence-server

server