Confluence cluster setup
Asked by admin:

Infra

* EFS (/share directory)
* 2 application aws nodes
* application loadbalancer
* postgresql (aws rds)

The above infra is setup in aws. Started with single node and then followed cluster setup document.

Node1 comes up and when trying to start node2 , node1 goes down with http 500 error.

If restarted service on node1 , node2 reports http 500 error.

node 2 (conf_access_log_

04.0.0.0 Safari/537.36  
\[17/Aug/2022:16:45:35 +0000\] admin http-nio-8090-exec-1 10.0.254.143 GET /index.action HTTP/1.1 200 14175ms 9316 [http://epo-ness-03b9c9.mvision-gpnes.local:8090/login.action?os_destination=%2Findex.action\&permissionViolation=true](http://epo-ness-03b9c9.mvision-gpnes.local:8090/login.action?os_destination=%2Findex.action&permissionViolation=true) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

node1(conf_access_log)

\[17/Aug/2022:17:17:16 +0000\] - http-nio-8090-exec-8 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0  
\[17/Aug/2022:17:17:16 +0000\] - http-nio-8090-exec-4 10.0.235.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0  
\[17/Aug/2022:17:17:36 +0000\] - http-nio-8090-exec-3 10.0.188.214 GET /status HTTP/1.1 500 1ms 17 - ELB-HealthChecker/2.0  
\[17/Aug/2022:17:17:36 +0000\] - http-nio-8090-exec-5 10.0.235.214 GET /status HTTP/1.1 500 1ms 17 - ELB-HealthChecker/2.0  
\[17/Aug/2022:17:17:56 +0000\] - http-nio-8090-exec-9 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0  
\[17/Aug/2022:17:17:56 +0000\] - http-nio-8090-exec-6 10.0.235.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0  
\[17/Aug/2022:17:18:16 +0000\] - http-nio-8090-exec-8 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0

node 1 (catalina.out)

... 70 more  
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!  
at com.hazelcast.instance.HazelcastInstanceProxy.getOriginal(HazelcastInstanceProxy.java:321)  
at com.hazelcast.instance.HazelcastInstanceProxy.getCluster(HazelcastInstanceProxy.java:219)  
at jdk.internal.reflect.GeneratedMethodAccessor115.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)  
at java.base/java.lang.reflect.Method.invoke(Unknown Source)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy83.getCluster(Unknown Source)  
at com.hazelcast.hibernate.HazelcastTimestamper.nextTimestamp(HazelcastTimestamper.java:38)  
at com.hazelcast.hibernate.AbstractHazelcastCacheRegionFactory.nextTimestamp(AbstractHazelcastCacheRegionFactory.java:76)  
at org.hibernate.internal.AbstractSharedSessionContract.\<init\>(AbstractSharedSessionContract.java:142)  
at org.hibernate.internal.AbstractSessionImpl.\<init\>(AbstractSessionImpl.java:29)  
at org.hibernate.internal.SessionImpl.\<init\>(SessionImpl.java:253)  
at org.hibernate.internal.SessionFactoryImpl$SessionBuilderImpl.openSession(SessionFactoryImpl.java:1229)  
at org.hibernate.internal.SessionFactoryImpl.openSession(SessionFactoryImpl.java:449)  
at com.atlassian.confluence.impl.core.persistence.hibernate.SwitchableCachingSessionFactory.openSession(SwitchableCachingSessionFactory.java:117)  
at jdk.internal.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)  
at java.base/java.lang.reflect.Method.invoke(Unknown Source)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy87.openSession(Unknown Source)  
at com.atlassian.hibernate.adapter.bridge.factory.SessionFactoryV2orV5BridgeProxy.openSession(SessionFacto

I need active active cluster. But 2 nodes are not active at same time.


Answers:
author: Kenneth Epker
body: When doing initial setup, which is best, selecting Non-clustered or clustered? When I enable it during setup I only get the 3 options, Multicast, TCP, and AWS. I'm assuming I need to enable clustering mode in the GUI. This is the only way I get the \<property name="confluence.cluster.join.type"\>\</property\> field in confluence.cfg.xml. Just having it enabled in my helm chart doesn't create the field in the cfg file. When I enabled cluster mode after setup and restart (delete k3s pod), the pod hangs and will not restart fully. I can exec into it and the confluence.cfg.xml is set back to "setupstart".


author: Kenneth Epker
body: The node is hanging again. First it has a problem connecting to hazelcast now it just hangs on Could not get swap target filter. Below is the log:

INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2  
INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2  
INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2  
INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.  
INFO:root:User is currently root. Will downgrade run user to confluence  
INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments \['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg'\]  
executing as current user  
If you encounter issues starting up Confluence, please see the Installation guide at <http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide>  
Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out  
---------------------------------------------------------------------------  
Using Java: /opt/java/openjdk/bin/java  
2023-10-26 20:05:17,439 INFO \[main\] \[atlassian.confluence.bootstrap.SynchronyProxyWatchdog\] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required  
---------------------------------------------------------------------------  
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED  
\[0.001s\]\[warning\]\[gc\] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-10-26_20-05-17.log instead.  
26-Oct-2023 20:05:18.122 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Connector\] failed to set property \[debug\] to \[0\]  
26-Oct-2023 20:05:18.130 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Connector\] failed to set property \[proxyPort\] to \[\]  
26-Oct-2023 20:05:18.138 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine\] failed to set property \[debug\] to \[0\]  
26-Oct-2023 20:05:18.142 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host\] failed to set property \[debug\] to \[0\]  
26-Oct-2023 20:05:18.202 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context\] failed to set property \[debug\] to \[0\]  
26-Oct-2023 20:05:18.278 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context/Valve\] failed to set property \[resolveHosts\] to \[false\]  
26-Oct-2023 20:05:18.280 WARNING \[main\] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match \[Server/Service/Engine/Host/Context\] failed to set property \[debug\] to \[0\]  
26-Oct-2023 20:05:18.621 INFO \[main\] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler \["http-nio-8090"\]  
26-Oct-2023 20:05:18.643 INFO \[main\] org.apache.catalina.startup.Catalina.load Server initialization in \[694\] milliseconds  
26-Oct-2023 20:05:18.679 INFO \[main\] org.apache.catalina.core.StandardService.startInternal Starting service \[Catalina\]  
26-Oct-2023 20:05:18.679 INFO \[main\] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: \[Apache Tomcat/9.0.73\]  
26-Oct-2023 20:05:21.093 INFO \[Catalina-utility-2\] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.  
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".  
SLF4J: Defaulting to no-operation (NOP) logger implementation  
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.  
26-Oct-2023 20:05:21.205 INFO \[Catalina-utility-2\] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath  
26-Oct-2023 20:05:21.581 INFO \[Catalina-utility-2\] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'  
2023-10-26 20:05:22,897 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] contextInitialized Starting Confluence 7.19.9 \[build 8804 based on commit hash d47376fcfbd30b32890091b1ace5511f4b0beb75\] - synchrony version 5.0.9  
2023-10-26 20:05:26,707 WARN \[Catalina-utility-1\] \[security.java8.serialfilter.DeserializationFilterConfigurator\] configure Class sun.misc.ObjectInputFilter not available in classpath. Not setting JDK 8 DeserializationFilter.  
java.lang.ClassNotFoundException: sun.misc.ObjectInputFilter  
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1412)  
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)  
at java.base/java.lang.Class.forName0(Native Method)  
at java.base/java.lang.Class.forName(Class.java:315)  
at com.atlassian.security.java8.serialfilter.DeserializationFilterConfigurator.configure(DeserializationFilterConfigurator.java:26)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.base/java.lang.reflect.Method.invoke(Method.java:566)  
at com.atlassian.confluence.setup.serial.ReflectiveFilterConfigurator.setReflectively(ReflectiveFilterConfigurator.java:11)  
at com.atlassian.confluence.setup.serial.JDKFilterConfigurator.setJDKSerialFilter(JDKFilterConfigurator.java:23)  
at com.atlassian.confluence.setup.DeserializationFilterConfigurator.configureDeserializationFilter(DeserializationFilterConfigurator.java:33)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:233)  
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)  
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)  
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)  
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)  
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)  
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.base/java.lang.Thread.run(Thread.java:829)  
2023-10-26 20:05:26,885 INFO \[Catalina-utility-1\] \[atlassian.confluence.cluster.DefaultClusterConfigurationHelper\] lambda$populateExistingClusterSetupConfig$5 Populating setup configuration if running with Cluster mode...  
2023-10-26 20:05:27,692 INFO \[Catalina-utility-1\] \[confluence.cluster.hazelcast.HazelcastClusterManager\] configure Configuring Hazelcast with instanceName \[confluence\], join configuration com.atlassian.confluence.cluster.KubernetesClusterJoinConfig@0, network interfaces \[10.42.2.52\] and local port 5701  
2023-10-26 20:05:27,693 INFO \[Catalina-utility-1\] \[confluence.cluster.hazelcast.HazelcastClusterManager\] startCluster Starting the cluster.  
2023-10-26 20:05:27,701 WARN \[Catalina-utility-1\] \[com.hazelcast.instance.HazelcastInstanceFactory\] log Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:  
--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED  
2023-10-26 20:05:30,381 INFO \[hz.confluence.event-4\] \[confluence.cluster.hazelcast.LoggingClusterMembershipListener\] memberAdded \[10.42.2.52\]:5701 joined the cluster  
2023-10-26 20:05:30,385 INFO \[hz.confluence.event-4\] \[confluence.cluster.hazelcast.LoggingClusterMembershipListener\] logClusterMembers Cluster now has 1 members: \[\[10.42.2.52\]:5701\]  
WARNING: An illegal reflective access operation has occurred  
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/hazelcast-3.12.13.jar) to field sun.nio.ch.SelectorImpl.selectedKeys  
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer  
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations  
WARNING: All illegal access operations will be denied in a future release  
2023-10-26 20:05:30,540 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[1\] retrying in 1 seconds...  
2023-10-26 20:05:32,043 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[2\] retrying in 2 seconds...  
2023-10-26 20:05:34,296 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[3\] retrying in 3 seconds...  
2023-10-26 20:05:37,676 WARN \[Catalina-utility-1\] \[spi.discovery.integration.DiscoveryService\] log \[10.42.2.52\]:5701 \[GCCS\] \[3.12.13\] Cannot fetch the current zone, ZONE_AWARE feature is disabled  
2023-10-26 20:05:37,679 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[1\] retrying in 1 seconds...  
2023-10-26 20:05:39,182 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[2\] retrying in 2 seconds...  
2023-10-26 20:05:41,436 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[3\] retrying in 3 seconds...  
2023-10-26 20:05:44,814 WARN \[Catalina-utility-1\] \[spi.discovery.integration.DiscoveryService\] log \[10.42.2.52\]:5701 \[GCCS\] \[3.12.13\] Cannot fetch name of the node, NODE_AWARE feature is disabled  
2023-10-26 20:05:44,824 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[1\] retrying in 1 seconds...  
2023-10-26 20:05:46,327 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[2\] retrying in 2 seconds...  
2023-10-26 20:05:48,580 WARN \[Catalina-utility-1\] \[com.hazelcast.kubernetes.RetryUtils\] log Couldn't discover Hazelcast members using Kubernetes API, \[3\] retrying in 3 seconds...  
2023-10-26 20:05:51,958 ERROR \[Catalina-utility-1\] \[internal.cluster.impl.DiscoveryJoiner\] log \[10.42.2.52\]:5701 \[GCCS\] \[3.12.13\] Failure in executing REST call  
com.hazelcast.kubernetes.RestClientException: Failure in executing REST call  
at com.hazelcast.kubernetes.RestClient.call(RestClient.java:114)  
at com.hazelcast.kubernetes.RestClient.get(RestClient.java:79)  
at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:503)  
at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:497)  
at com.hazelcast.kubernetes.RetryUtils.retry(RetryUtils.java:53)  
at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:497)  
at com.hazelcast.kubernetes.KubernetesClient.endpoints(KubernetesClient.java:80)  
at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:70)  
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:135)  
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:71)  
at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:70)  
at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:59)  
at com.hazelcast.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:151)  
at com.hazelcast.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:111)  
at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)  
at com.hazelcast.instance.Node.join(Node.java:820)  
at com.hazelcast.instance.Node.start(Node.java:455)  
at com.hazelcast.instance.HazelcastInstanceImpl.\<init\>(HazelcastInstanceImpl.java:136)  
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)  
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)  
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)  
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)  
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.createHazelcastInstance(HazelcastClusterManager.java:390)  
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:359)  
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:331)  
at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:477)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:875)  
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:236)  
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)  
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)  
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)  
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)  
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)  
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.base/java.lang.Thread.run(Thread.java:829)  
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 404 Not Found"  
at java.base/sun.net.[www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2177](http://www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2177))  
at java.base/sun.net.[www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:199](http://www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:199))  
at java.base/sun.net.[www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592](http://www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592))  
at java.base/sun.net.[www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520](http://www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520))  
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)  
at java.base/sun.net.[www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334](http://www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334))  
at com.hazelcast.kubernetes.RestClient.checkHttpOk(RestClient.java:131)  
at com.hazelcast.kubernetes.RestClient.call(RestClient.java:111)  
... 42 more  
2023-10-26 20:05:51,960 ERROR \[Catalina-utility-1\] \[com.hazelcast.instance.Node\] log \[10.42.2.52\]:5701 \[GCCS\] \[3.12.13\] Could not join cluster. Shutting down now!  
2023-10-26 20:05:51,965 WARN \[Catalina-utility-1\] \[com.hazelcast.instance.Node\] log \[10.42.2.52\]:5701 \[GCCS\] \[3.12.13\] Terminating forcefully...  
2023-10-26 20:05:52,024 WARN \[Catalina-utility-1\] \[com.hazelcast.util.PhoneHome\] log \[10.42.2.52\]:5701 \[GCCS\] \[3.12.13\] Could not schedule phone home task! Most probably Hazelcast failed to start.  
2023-10-26 20:05:52,030 ERROR \[Catalina-utility-1\] \[atlassian.confluence.setup.ConfluenceConfigurationListener\] contextInitialized An error was encountered while bootstrapping Confluence (see below):  
Node failed to start!  
java.lang.IllegalStateException: Node failed to start!  
at com.hazelcast.instance.HazelcastInstanceImpl.\<init\>(HazelcastInstanceImpl.java:138)  
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)  
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)  
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)  
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)  
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.createHazelcastInstance(HazelcastClusterManager.java:390)  
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:359)  
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:331)  
at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:477)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:875)  
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)  
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:236)  
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)  
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)  
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)  
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)  
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)  
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)  
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)  
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  
at java.base/java.lang.Thread.run(Thread.java:829)  
2023-10-26 20:05:52,044 WARN \[Catalina-utility-1\] \[atlassian.confluence.plugin.PluginFrameworkContextListener\] contextInitialized Not starting full plugin system due to upgrade  
2023-10-26 20:05:52,048 \[Catalina-utility-1\]  
\[Filter: profiling\] defaulting to off \[autostart=false\]  
2023-10-26 20:05:52,611 INFO \[Catalina-utility-1\] \[com.atlassian.confluence.lifecycle\] init Confluence is ready to serve  
26-Oct-2023 20:05:52.622 INFO \[main\] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler \["http-nio-8090"\]  
26-Oct-2023 20:05:52.633 INFO \[main\] org.apache.catalina.startup.Catalina.start Server startup in \[33989\] milliseconds  
2023-10-26 20:05:52,850 ERROR \[http-nio-8090-exec-6\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00f905abbaef89a3  
2023-10-26 20:05:52,850 ERROR \[http-nio-8090-exec-5\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00cfdde337b8ea1f  
2023-10-26 20:05:52,850 ERROR \[http-nio-8090-exec-2\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00b5f43c1e4d179b  
2023-10-26 20:05:52,850 ERROR \[http-nio-8090-exec-3\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00fdf37be9d5e116  
2023-10-26 20:05:52,850 ERROR \[http-nio-8090-exec-4\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00cd34bc1d90f324  
2023-10-26 20:05:52,850 ERROR \[http-nio-8090-exec-1\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0036ab7a65b0d28d  
2023-10-26 20:05:54,641 ERROR \[http-nio-8090-exec-7\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 007cf397d2d52c3e  
2023-10-26 20:05:56,537 ERROR \[http-nio-8090-exec-8\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 002231c67d1d2bf6  
2023-10-26 20:06:01,537 ERROR \[http-nio-8090-exec-9\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 006b51b8421d7f92  
2023-10-26 20:06:06,537 ERROR \[http-nio-8090-exec-10\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0079708036e2deb2  
2023-10-26 20:06:11,537 ERROR \[http-nio-8090-exec-4\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00c4ce8d94a4cb15  
2023-10-26 20:06:16,537 ERROR \[http-nio-8090-exec-6\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00d4f824d6ee5c34  
2023-10-26 20:06:21,538 ERROR \[http-nio-8090-exec-3\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00e02b3e2791ca10  
2023-10-26 20:06:26,538 ERROR \[http-nio-8090-exec-2\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 007f211bac0484ab  
2023-10-26 20:06:30,641 ERROR \[http-nio-8090-exec-5\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 003eb0d735b8a9b9  
2023-10-26 20:06:31,537 ERROR \[http-nio-8090-exec-1\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00349d54be73d4a9  
2023-10-26 20:06:36,538 ERROR \[http-nio-8090-exec-7\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00ed58d7fbbce6a4  
2023-10-26 20:06:41,537 ERROR \[http-nio-8090-exec-8\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 006295c06a78e381  
2023-10-26 20:06:46,538 ERROR \[http-nio-8090-exec-9\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0049c161e3d98036  
2023-10-26 20:06:51,537 ERROR \[http-nio-8090-exec-10\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0063cdff7f81f37e  
2023-10-26 20:06:56,537 ERROR \[http-nio-8090-exec-4\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00feaa7a78b34057  
2023-10-26 20:07:01,537 ERROR \[http-nio-8090-exec-6\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00316ade4270a539  
2023-10-26 20:07:06,537 ERROR \[http-nio-8090-exec-3\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0092ca6137da0bbc  
2023-10-26 20:07:11,537 ERROR \[http-nio-8090-exec-2\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0088a5d4c8e90f00  
2023-10-26 20:07:13,641 ERROR \[http-nio-8090-exec-5\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0065eb50c467ed5b  
2023-10-26 20:07:16,537 ERROR \[http-nio-8090-exec-1\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 0004d03f9aaf3c2c  
2023-10-26 20:07:21,537 ERROR \[http-nio-8090-exec-7\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 001608a488109f31  
2023-10-26 20:07:26,537 ERROR \[http-nio-8090-exec-8\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 008dd5b05b06e199  
2023-10-26 20:07:31,537 ERROR \[http-nio-8090-exec-9\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00d23a5d47c00ed7  
2023-10-26 20:07:36,538 ERROR \[http-nio-8090-exec-10\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 000f545e96744239  
2023-10-26 20:07:41,537 ERROR \[http-nio-8090-exec-4\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00ca976e2efcce9c  
2023-10-26 20:07:46,536 ERROR \[http-nio-8090-exec-6\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 003f2ca304da2db6  
2023-10-26 20:07:47,641 ERROR \[http-nio-8090-exec-3\] \[atlassian.confluence.util.AbstractBootstrapHotSwappingFilter\] doFilter Could not get swap target filter  
-- url: /status \| traceId: 00f0e2c2479d6c77


author: Yevhen
body: I'm sorry - helm upgrade


author: Kenneth Epker
body: Not familiar with the step **Run help upgrade**. How do I do that?


author: Kenneth Epker
body: Great, trying it now.


author: Yevhen
body: Correct, none of them will. All you need to do is to set confluence.clustering to true, Helm chart will do the rest. You said you migrated your local-home and I think that's the problem.

Can you please  

1. Scale down to 0  
2. set:  

\`\`\`

confluence:  
additionalEnvironmentVariables:  
- name: ATL_FORCE_CFG_UPDATE

value: "true"

\`\`\`  
This will overwrite cfg.xml in local home and set the right discovery type.  
3. Run help upgrade  
4. Make sure \<property name="confluence.cluster.join.type"\>kubernetes\</property\> is in confluence.cfg.xml in local home  
5. Scale to 2  

<br />


author: Kenneth Epker
body: I'm sorry, didn't realize we changed our images to pull the docker.io/atlassian/confluence image via helm repo atlassian-github (v1.13.0). Looking through these charts though, I only see clustering connection types: Multicast, TCP/IP or AWS. None of these connection types will work in my kubernetes environment.


author: Yevhen
body: I am not familiar with IronBank Helm Charts. I do recommend using the official Helm charts (you can still use IronBank Confluence image though)


author: Kenneth Epker
body: I'm using the dso Ironbank helm chart to deploy my confluence DC product. I do not see any option to modify the confluence.cfg.xml file ahead of time to use "kubernetes" connection type. Do you have any suggestions?


author: Kenneth Epker
body: Yes, I did enable clustering. I've copied the contents of local-home to shared-home and made sure the second node received a copy also, especially the confluence.cfg.xml. I've tried both Multicast and TCP/IP as the clustering type with no luck. I just came across a blog that mentions a Kubernetes connection type for clustering. Trying that now.


author: Yevhen
body: [@Kenneth Epker](/t5/user/viewprofilepage/user-id/5277346) did you enable clustering in values.yaml?

\>Configured first node for clustering

How did you do that?


author: Kenneth Epker
body: I have the exact same error. I'm running confluence through Kubernetes. Configured first node for clustering, no problems, shows up in the gui under confluence administration-\> clustering as active. Increased replicas to 2, node 2 comes up and node 1 goes inactive with the errors described above. Now node 2 shows up in clustering and node 1 is no longer listed.


Tags: confluence-server

server