SpringMVC dispatcher could not be started (Fatal error has ocurred)
Asked by Joe Powell:

Hi, we have bitbucket 8.1.1 running on CentOS 7.9.

It did run ok until the last update of these 2 packages:git-1.8.3.1-23.el7_8 \& perl-Git-1.8.3.1-23.el7_8 to git-1.8.3.1-23.el7_9 \& perl-Git-1.8.3.1-23.el7_9.

Since bitbucket refuse to run with "*Web server failed to start. Port 7990 was already in use."*

The log file shows this:

*2023-03-07 14:46:57,633 INFO \[main\] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v8.1.1 using Java 1.8.0_322 on engtools-rep with PID 4630 (/opt/atlassian/bitbucket/8.1.1/app/WEB-INF/classes started by root in /opt/atlassian/bitbucket/8.1.1/bin)*   
*2023-03-07 14:46:57,639 INFO \[main\] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to 1 default profile: "default"*   
*2023-03-07 14:46:58,626 INFO \[main\] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 8.1.1 (38bec1c built on Thu Jun 09 10:38:45 BST 2022)*   
*2023-03-07 14:46:58,626 INFO \[main\] c.a.b.i.boot.log.BuildInfoLogger JVM: Temurin OpenJDK 64-Bit Server VM 1.8.0_322-b06*   
*2023-03-07 14:47:00,043 WARN \[main\] o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.PortInUseException: Port 7990 is already in use*   
*2023-03-07 14:47:00,043 INFO \[main\] c.a.b.i.boot.log.BuildInfoLogger Bitbucket 8.1.1 has shut down*   
*2023-03-07 14:47:00,530 WARN \[main\] o.a.c.loader.WebappClassLoaderBase The web application \[ROOT\] appears to have started a thread named \[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser\] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:*   
*sun.management.NotificationEmitterSupport.removeNotificationListener(NotificationEmitterSupport.java:109)*   
*oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaserListener.unregister(BlockSource.java:413)*   
*oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:335)*   
*2023-03-07 14:47:00,603 ERROR \[main\] o.s.b.d.LoggingFailureAnalysisReporter*

*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\**   
*APPLICATION FAILED TO START*   
*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\**

*Description:*

*Web server failed to start. Port 7990 was already in use.*

*Action:*

*Identify and stop the process that's listening on port 7990 or configure this application to listen on another port.*

*2023-03-07 14:47:18,525 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[hazelcastFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,527 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,527 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[entityMovedFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,528 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[pluginProvidedAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,528 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[trustedApplicationsAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,528 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[goGetFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,529 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,529 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[entityMovedFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,529 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[trustedApplicationsAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,529 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[springSecurityFilterChain\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,529 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[authenticationAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,529 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[rateLimitFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,533 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[standardResponseHeadersFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,533 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,534 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[entityMovedFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,534 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[trustedApplicationsAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,534 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[springSecurityFilterChain\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,535 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[authenticationAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,535 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[rateLimitFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:18,535 WARN \[http-nio-7990-exec-9\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[standardResponseHeadersFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,968 WARN \[http-nio-7990-exec-10\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[hazelcastFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,969 WARN \[http-nio-7990-exec-10\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,983 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[hazelcastFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[entityMovedFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[pluginProvidedAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[trustedApplicationsAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[goGetFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[entityMovedFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[trustedApplicationsAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[springSecurityFilterChain\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,984 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[authenticationAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,986 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[rateLimitFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,986 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[standardResponseHeadersFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[requestAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[entityMovedFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[trustedApplicationsAuthenticationFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[springSecurityFilterChain\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[authenticationAttributeFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[rateLimitFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 14:47:23,987 WARN \[http-nio-7990-exec-1\] c.a.j.s.w.f.BypassableDelegatingFilterProxy Bypassing \[standardResponseHeadersFilter\]; no Spring WebApplicationContext is available*   
*2023-03-07 15:17:39,625 WARN \[Catalina-utility-1\] com.hazelcast.web.SessionListener The com.hazelcast.web.WebFilter could not be found. com.hazelcast.web.SessionListener should be paired with a com.hazelcast.web.WebFilter.*

I tried downgrading the packages with the same resolution. It seems to be a configuration issue but I can't see how the update could have affected it.

Any idea how to resolve this problem?

I'm grateful for any suggestion.


Answers:
author: Christian Glockner
body: Hi Joe,

If port 7990 is already in use it probably just means that there's a lingering java process from Bitbucket Server that is holding on to that part.  

I'd check using **netstat -ltpn \| grep 7990** which process is holding on to that port, then stop it and then start Bitbucket Server again.

Cheers,

Christian

Premier Support Engineer

Atlassian

comments: author: Joe Powell
body: Hi Christian,

Thanks for you response.

Yes that was on of the first thing I tried without success.

I tried it again today with following oputput:

*netstat -ltnp \| grep java*   
*tcp6 0 0 :::8090 :::\* LISTEN 2827/java*   
*tcp6 0 0 :::8091 :::\* LISTEN 3277/java*   
*tcp6 0 0 127.0.0.1:8000 :::\* LISTEN 2827/java*   
*tcp6 0 0 127.0.0.1:8005 :::\* LISTEN 1490/java*   
*tcp6 0 0 :::8080 :::\* LISTEN 1490/java*   
*\[root@engtools-rep \~\]# cd /opt/atlassian/bitbucket/8.1.1/bin/*   
*\[root@engtools-rep bin\]# ./start-bitbucket.sh*   
*Starting Atlassian Bitbucket as the current user*

*Starting the bundled search server*   
*Hint: Run start-bitbucket.sh --no-search to skip starting the bundled search server*   
*future versions of OpenSearch will require Java 11; your Java version from \[/opt/atlassian/bitbucket/8.1.1/jre\] does not meet this requirement*   
*future versions of OpenSearch will require Java 11; your Java version from \[/opt/atlassian/bitbucket/8.1.1/jre\] does not meet this requirement*   
*Bundled search server started successfully*

*Bitbucket is being run with a umask that contains potentially unsafe settings.*   
*The following issues were found with the mask "u=rwx,g=rx,o=rx" (0022):*   
*- Access is allowed to 'others'. It is recommended that 'others' be denied*   
*all access for security reasons.*   
*The recommended umask for Bitbucket is "u=,g=w,o=rwx" (0027) and can be*   
*configured in _start-webapp.sh*

*The current open files limit is set to less than 6192*   
*Attempting to increase limit...*   
*Limit increased to 6192 open files*

*Starting Bitbucket webapp at <http://localhost:7990>*   
*The Bitbucket webapp has been started.*

*If you cannot access Bitbucket within 3 minutes, or encounter other issues, check the troubleshooting guide at:*   
*<https://go.atlassian.com/bbs-troubleshooting-installations>*   
*\[root@engtools-rep bin\]# uncaught exception in thread \[main\]*   
*java.lang.RuntimeException: can not run opensearch as root*   
*at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124)*   
*at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)*   
*at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)*   
*at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)*   
*at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)*   
*at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)*   
*at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)*   
*at org.opensearch.cli.Command.main(Command.java:101)*   
*at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)*   
*at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)*   
*For complete error details, refer to the log at /var/atlassian/application-data/bitbucket/log/search/bitbucket_search.log*

I found in your webpage similar problem in case 5 "<https://confluence.atlassian.com/bitbucketserverkb/springmvc-dispatcher-could-not-be-started-779171477.html>"

With following resolution:

1. ***Resolution for cause #5:**The error message states unsatisfied dependancy exception for GIT version on the Bitbucket server. On Some of the latest linux distributions GIT version is 2.38.x or later, which is still not supported on Bitbucket 8.6.x or below and will likely Bitbucket fails to start. Please refer Supported platform page for your Bitbucket version and compile a supported version of GIT to resolve the issue.*
2. When I revert the changes via snapshot, it works fine. If I only undo the package updates it stay broken.
3. Another thing I can't understand and it might not be relevant, is the message about*OpenSearch will require Java 11; your Java version from \[/opt/atlassian/bitbucket/8.1.1/jre\] does not meet this requirement future versions of OpenSearch will require Java 11.*

Thanks for your support.

Joe.


author: Joe Powell
body: Hi Christian,

just add to another attempt to sort out this problem, I stop all java processes ( Bitbucket,Jira,Confluence ) and start bitbucker again. Now I check the java process to make sure it was running and got this result: " netstat -tlpn \| grep java  
tcp6 0 0 :::7990 :::\* LISTEN 2846/java  
tcp6 0 0 :::5701 :::\* LISTEN 2846/java "

If I stop 5701, automatically close 7990.

Could that be part of the problem?

Thanks,


author: Christian Glockner
body: Hi Joe,

5701 belongs to the Hazelcast part of Bitbucket Server, so no wonder it shuts down if you close that port.

From the earlier messages it seems like Bitbucket Server actually starts fine, but it's the Opensearch part that fails to start because you're starting it as root (which is not a good idea for Bitbucket Server too, by the way). Try starting it as a dedicated user (the default username is atlbitbucket) to see if that makes a difference.

Cheers,

Christian

Premier Support Engineer

Atlassian


Tags: bitbucket-server

server