Repository: spring-projects/spring-framework

Issue #22332: MongoWaitQueueFullException in Webflux + Reactive Mongo
Created: 2019-02-01T16:08:31Z
Affects: 5.1.2.RELEASE


The latest  TechEmpower's benchmark shows a lot of errors for Webflux + Spring data mongo.
I digged and found out that under load, we get a lot of
com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded. 
exceptions.
From our discussion on gitter, @mp911de thinks that "WebFlux could have a notion of connection backpressure and should limit subscription rate/demand rate to provide a sense of protection for downstream components within the application".
Code of the benchmark here
Related issue : #20338

Duplicate issue #22775: com.mongodb.MongoWaitQueueFullException with Webflux + reactive Mongo driver
Created: 2019-04-09T13:13:21Z
I am trying to do load test of a simple Spring webflux application using gatling tool. Application is developed using 'spring-boot-starter-webflux' and 'spring-boot-starter-data-mongodb-reactive' projects. It simply read mongo document with a specific unique column. I inject concurrent users using gatling setUp(scn.inject(atOnceUsers(userCount)).protocols(httpConf))
I start mongo db instance like shown below
Replica
mongod --replSet rs0 --port 27020 --bind_ip localhost,somehostname,some_ip --dbpath C:\mongo\data\db0 --smallfiles --oplogSize 128 mongod --replSet rs0 --port 27021 --bind_ip localhost,somehostname,some_ip --dbpath C:\mongo\data\db1 --smallfiles --oplogSize 128 mongod --replSet rs0 --port 27022 --bind_ip localhost,somehostname,some_ip --dbpath C:\mongo\data\db2 --smallfiles --oplogSize 128

standalone
mongod --port 27018 --bind_ip localhost,somehostname,some_ip --dbpath C:\mongo\data\db9 --smallfiles --oplogSize 128

Setup 1: application running in Windows Desktop(Intel i5 and 16GB RAM), Mongo DB replica mode (3 node) running on windows laptop (Intel i7 processor and 16GB RAM) and Gatling load test scripts also on desktop. Both application and Gatling scripts on desktop are containerized. Queue size is 500 by default, I have overridden with 1000 queue size using waitQueueMultiple
I am getting this com.mongodb.MongoWaitQueueFullException: with even 3000 concurrency itself.
Setup 2: I have same setup as above but Mongo DB is running in standalone mode
I am getting this com.mongodb.MongoWaitQueueFullException: with even 3000 concurrency itself.
Setup 3: application, Mongo DB standalone mode and Gatling load test scripts all running on Desktop. And all are containerized and connected with a bridge network Queue size is 500 by default, overridden with 1000 queue size this setup works fine till 10000 concurrency. I understand here there is no role of network latency , so the better performance
I have below question


How to resolve this exception , apart from increasing the queue size .


Why there is a Mongo DB performance difference between Standalone and Replica mode
As mentioned above setup 1 and 2 raises exception when concurrency ~3000 users. I repeat these tests at random time of the day. But at some point of time standalone setup of mongo database performance extremely well and till 48000 concurrent users application scales well(no exception thrown). I checked that Mongo is receivng the requests by watching mongostat/mongotop and also confirmed by running db.adminCommand("top") before and after the each test. I can confirm that read counts are increasing by number of concurrent users i used for test. My only worry is same time if I use mongo DB in replica mode (Setup 1) it does not show the better performance , it continues to throw exception at 3000 concurrency itself. Why replica mode is not performaing equal to standalone mode ) Application code is same , gatling script is same.


I am using mongodb-driver-reactivestreams 1.9.2 Mongo Database 4.0.8

Closing comment (2019-04-09T19:57:18Z): This is related to #22332 so see discussion there. Can you provide some information about what controller methods do so we can understand how many queries are made per request, etc?  /cc @mp911de