Hazelcast instance is not active!
Asked by Oday Rafeh:

Hi There,

We just run into an issue with our software and this error coming directly to all users its very important to solve this as soon as possible because no one can use the software right now.

HTTP Status 500 -- Internal Server Error {#toc-hId--552851114}
--------------------------------------------------------------

*** ** * ** ***

**Type** Exception Report

**Message** Hazelcast instance is not active!

**Description** The server encountered an unexpected condition that prevented it from fulfilling the request.

**Exception**

```
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!	com.hazelcast.instance.DefaultNodeExtension$1.get(DefaultNodeExtension.java:205)	com.hazelcast.instance.DefaultNodeExtension$1.get(DefaultNodeExtension.java:202)	com.hazelcast.internal.serialization.impl.bufferpool.BufferPoolThreadLocal.get(BufferPoolThreadLocal.java:107)	com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:141)	com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133)	com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118)	com.hazelcast.map.impl.proxy.MapProxySupport.toDataWithStrategy(MapProxySupport.java:1262)	com.hazelcast.map.impl.proxy.NearCachedMapProxyImpl.getInternal(NearCachedMapProxyImpl.java:113)	com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120)	com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:308)	com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:125)	com.atlassian.confluence.cache.DefaultConfluenceCache.get(DefaultConfluenceCache.java:53)	com.atlassian.confluence.cache.ConfluenceMonitoringCache.get(ConfluenceMonitoringCache.java:57)	com.atlassian.confluence.impl.cache.ReadThroughAtlassianCache.get(ReadThroughAtlassianCache.java:88)	com.atlassian.confluence.impl.cache.ReadThroughCache.get(ReadThroughCache.java:28)	com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.retrieve(ReadThroughCachingBandanaPersister.java:82)	com.atlassian.confluence.impl.bandana.ReadThroughCachingBandanaPersister.retrieve(ReadThroughCachingBandanaPersister.java:75)	com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)	com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)	com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:40)	com.atlassian.confluence.languages.DefaultLocaleManager.localeValidAndSaveable(DefaultLocaleManager.java:279)	com.atlassian.confluence.languages.DefaultLocaleManager.setRequestLanguages(DefaultLocaleManager.java:267)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.lambda$null$2(LanguageExtractionFilter.java:37)	java.base/java.util.Optional.ifPresent(Unknown Source)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.lambda$doFilter$3(LanguageExtractionFilter.java:37)	java.base/java.util.Optional.ifPresent(Unknown Source)	com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:37)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)	com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84)	com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68)	com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:43)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:67)	com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:59)	com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)	com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)	com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)	com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)	com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)	com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:42)	com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
```

**Note** The full stack trace of the root cause is available in the server logs.


Answers:
author: G subramanyam
body: Hi [@Oday Rafeh](/t5/user/viewprofilepage/user-id/4819196) welcome to the Atlassian community.

I have found this **[thread](https://community.atlassian.com/t5/Bitbucket-questions/what-causes-Hazelcast-instance-to-become-inactive/qaq-p/80060)** similar to your issue. Beside, I would request you to log a request with **[Atlassian support team](https://support.atlassian.com/contact/?postauth=true#/)**. under technical issues \& bugs.

comments: author: Oday Rafeh
body: Thanks for your Help

I found that the directory

/opt/Atlassian/confluence/bin/

does not exist and when I tried to copy this directory from the other working node the error is gone but now confluence has not started and showing this error

Unit confluence.service has begun starting up.  
Jun 15 22:45:15 conf01 systemd\[29749\]: confluence.service: Failed to execute command: Permission denied  
Jun 15 22:45:15 conf01 systemd\[29749\]: confluence.service: Failed at step EXEC spawning /opt/atlassian/confluence/bin-- Subject: Process /opt/atlassian/confluence/bin/start-confluence.sh could not be executed  
-- Defined-By: systemd

Process /opt/atlassian/confluence/bin/start-confluence.sh could not be executed


author: Vikrant Yadav
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/4819196) How you fixed this issue? I am also facing the same issue. Kindly suggest.


author: Oday Rafeh
body: Hi, [@Vikrant Yadav](/t5/user/viewprofilepage/user-id/1801711) do you still facing this issue?


author: Vikrant Yadav
body: Hi [@Oday Rafeh](/t5/user/viewprofilepage/user-id/4819196) Thanks for the reply. But issue is fixed now.


author: Ilkay Tarik Cansiz
body: I am having the same issue with deploying confluence data centre using helm chart, currently confluence is running on open shift cluster and it seems like multicast discovery is not available and we can not create cluster role and cluster role binding since there are other running on the cluster do you have any suggestions?  
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!  
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)  
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)  
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)  
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)  
at jdk.internal.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.base/java.lang.reflect.Method.invoke(Method.java:566)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy79.getTransaction(Unknown Source)  
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)  
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)  
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)  
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)  
at com.sun.proxy.$Proxy87.retrieve(Unknown Source)  
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)  
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)  
at com.atlassian.confluence.impl.settings.BandanaGlobalSettingsManager.getGlobalSettings(BandanaGlobalSettingsManager.java:43)  
at com.atlassian.confluence.impl.settings.DelegatingLegacySettingsManager.getGlobalSettings(DelegatingLegacySettingsManager.java:57)  
at com.atlassian.confluence.status.service.DefaultSystemInformationService.isShowInfoOn500(DefaultSystemInformationService.java:326)  
at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:406)  
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)  
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)  
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)  
... 66 more  
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!  
at com.hazelcast.instance.HazelcastInstanceProxy.getOriginal(HazelcastInstanceProxy.java:321)  
at com.hazelcast.instance.HazelcastInstanceProxy.getCluster(HazelcastInstanceProxy.java:219)  
at jdk.internal.reflect.GeneratedMethodAccessor172.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.base/java.lang.reflect.Method.invoke(Method.java:566)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy73.getCluster(Unknown Source)  
at com.hazelcast.hibernate.HazelcastTimestamper.nextTimestamp(HazelcastTimestamper.java:38)  
at com.hazelcast.hibernate.AbstractHazelcastCacheRegionFactory.nextTimestamp(AbstractHazelcastCacheRegionFactory.java:76)  
at org.hibernate.internal.AbstractSharedSessionContract.\<init\>(AbstractSharedSessionContract.java:142)  
at org.hibernate.internal.AbstractSessionImpl.\<init\>(AbstractSessionImpl.java:29)  
at org.hibernate.internal.SessionImpl.\<init\>(SessionImpl.java:253)  
at org.hibernate.internal.SessionFactoryImpl$SessionBuilderImpl.openSession(SessionFactoryImpl.java:1229)  
at org.hibernate.internal.SessionFactoryImpl.openSession(SessionFactoryImpl.java:449)  
at com.atlassian.confluence.impl.core.persistence.hibernate.SwitchableCachingSessionFactory.openSession(SwitchableCachingSessionFactory.java:117)  
at jdk.internal.reflect.GeneratedMethodAccessor252.invoke(Unknown Source)  
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.base/java.lang.reflect.Method.invoke(Method.java:566)  
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)  
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)  
at com.sun.proxy.$Proxy80.openSession(Unknown Source)  
at com.atlassian.hibernate.adapter.bridge.factory.SessionFactoryV2orV5BridgeProxy.openSession(SessionFactoryV2orV5BridgeProxy.java:95)  
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:413)


Tags: atl-gruland

cloud

confluence-server

server