SpringMVC dispatcher [springMvc] could not be started
Asked by Holly Marsh:

Upgrading from 7.21.4 to 8.5.4. Using git 2.35.3, and java 17.0.3. Starting up the first node in my Data Center cluster, I'm seeing this in the logs:  

**launcher.log**

\[main\] DEBUG com.atlassian.bitbucket.internal.boot.security.JmxDeserializationFilterReflectionRegister - Running on Java 9+ using java.io.ObjectInputFilter

**atlassian-bitbucket.log**
> 2023-09-23 09:21:16,667 ERROR \[hz.hazelcast.event-1\] c.h.s.i.e.impl.EventServiceImpl \[10.2.39.133\]:5701 \[bitbucketProd\] \[3.12.13\] hz.hazelcast.event-1 caught an exception while processing:com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher@1cb6e277  
> com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.InvalidClassException: filter status: REJECTED  
> at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:70)  
> at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:193)  
> at com.hazelcast.map.impl.DataAwareEntryEvent.getValue(DataAwareEntryEvent.java:90)  
> at com.atlassian.cache.hazelcast.HazelcastCacheManager.lambda$new$1(HazelcastCacheManager.java:106)  
> at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:67)  
> at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:64)  
> at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)  
> at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)  
> at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:105)  
> at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:78)  
> at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)  
> at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:244)  
> at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:227)  
> Caused by: java.io.InvalidClassException: filter status: REJECTED  
> at java.base/java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1414)  
> at java.base/java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1868)  
> at java.base/java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1821)  
> at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:839)  
> at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:988)  
> at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2034)  
> at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1909)  
> at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2235)  
> at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744)  
> at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:514)  
> at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:472)  
> at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:84)  
> at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77)  
> at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)  
> at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)  
> ... 15 common frames omitted  
> Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make public java.lang.Class java.io.ObjectInputStream$FilterValues.serialClass() accessible: module java.base does not "opens java.io" to unnamed module @2f112965  
> at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)  
> at com.atlassian.bitbucket.internal.boot.security.JmxDeserializationFilterReflectionRegister$DeserializationFilterInvocationHandler.invoke(JmxDeserializationFilterReflectionRegister.java:87)  
> at jdk.proxy1/jdk.proxy1.$Proxy0.checkInput(Unknown Source)  
> at java.base/java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1383)  
> at java.base/java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1868)  
> at java.base/java.io.ObjectInputStream.readTypeString(ObjectInputStream.java:1821)  
> at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:839)  
> at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:988)  
> at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2034)  
> at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1909)  
> at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2235)  
> at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744)  
> at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:514)  
> at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:472)  
> at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:84)  
> at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77)  
> at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)  
> at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)  
> at com.hazelcast.map.impl.DataAwareEntryEvent.getValue(DataAwareEntryEvent.java:90)  
> at com.atlassian.cache.hazelcast.HazelcastCacheManager.lambda$new$1(HazelcastCacheManager.java:106)  
> at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:67)  
> at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:64)  
> at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)  
> at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)  
> at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)  
> at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:105)  
> at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:78)  
> at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)  
> at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:244)  
> at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:227)  
> ... 3 frames trimmed


Answers:
author: Kai G?nster
body: We had exactly the same problem with a fresh install of Bitbucket 8.9.x using Java 17, running on Debian Bookworm. We found a solution, but I must emphasize that it's not a great solution.

To make it work, extend your JAVA_OPTS environment as follows:

```
HAZELCAST_OPTS=" --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL- UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMEDi --add-opens java.base/java.io=ALL-UNNAMED"

JAVA_OPTS=" $JAVA_OPTS $HAZELCAST_OPTS"
```

Your Bitbucket instance should then come up and work.

Now, why is that not a great solution? What these settings essentially do is disable some Java security features that were added in Java 9. Without going into too much detail, before Java 9 and library and plugin could access all the JDK core classes, with Java 9 this was restricted through the use of a new module system that can deny or allow this access. What these settings do is open a lot of core modules again to be used by anyone, probably including Bitbucket plugins.

This exposes you to the same kind of risks that you used to have with Java 8 and earlier anyway, but they are risks you could avoid now.

I would consider this problem a Bitbucket bug, IMO the start scripts should set the necessary module permissions, and only those, not the ALLOW ALL TO ALL approach I'm taking here.

comments: author: Kai G?nster
body: Addendum: with the above solution, our BitBucket instance came up, but then we got problems with LDAP authentication also due to the Java module system. to fix those, we had to extend HAZELCAST_OPTS in the code above to include

```
 --add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED
```

I guess at that point it's not technically accurate anymore to call it HAZELCAST_OPTS, but meh.


Tags: bitbucket-server

server