MongoDB Performance 3.4 vs 4.4 vs 5.0 vs 6.0
Tags: mongodb, database, performance, devops
Published: 2022-09-26 13:41:01

We've been using MongoDB v3.4.24 for quite a long time. And ever since MongoDB v6 got released, we have wanted to benchmark all these new versions to see if they fit our needs. [We did such a benchmark](https://medium.com/@hartator/mongodb-4-2-vs-4-0-3-6-3-4-and-3-2-benchmarks-ee96a09ef231) years ago against MongoDB v4, but it turned out that MongoDB v4 was worse than MongoDB v3 in terms of performance. This time, we'll do a more detailed benchmark on MongoDB 3.4.24, 4.4.16, 5.0.12, and 6.0.1.

We ran the benchmark with https://github.com/mongodb/mongo-perf.

To better observe system resource usage and minimize the noises and randomness of the results, we've separated the testing server and client into 2 VMs with dedicated CPUs:

<table>
<tbody>
<tr>
<td>Server</td>
<td>DigitalOcean CPU-Optimized 4 CPUs / 8 GB, Ubuntu 20.04</td>
</tr>
<tr>
<td>Client</td>
<td>DigitalOcean CPU-Optimized 2 CPUs / 4 GB, Ubuntu 20.04 + MongoDB Shell (v5.0.12) + mongo-perf</td>
</tr>
</tbody>
</table>

## Server benchmark

Here is the python script for running the benchmark (change `6.0.1` to the version that is being tested, change `10.1.0.2` to the internal IP of the testing server, and change `-t 1` to `-t 4` to run with four threads):

```python
#!/usr/bin/bash
tests=("simple_insert" "simple_multi_update" "simple_query" "simple_remove" "simple_text" "simple_update" "simple_update_mms" "simple_geo")
for i in {1..10}; do
  echo "==== Round $i ===="
  for t in "${tests[@]}"; do
    python3 -u benchrun.py -f testcases/$t.js --host 10.1.0.2 -t 1 --trialTime 1 --out 6.0.1-$t-$i.json
  done
done
```

We ran the tests ten times for each server, cut out the lowest and the highest value, and averaged the rest. We didn't run through all tests provided by the repo, but that's enough for our use case.

Here are the results for one thread `-t 1` (higher is better, the winner for each test is highlighted in red):

![Mongodb Benchmark 1 thread](https://serpapi.com/blog/content/images/2022/09/mongo-bench-1.png)

[The results for the four threads](https://serpapi.com/blog/content/images/2022/09/mongo-bench-4.png) `-t 4` look similar on winner distributions.

We are also attaching the system resource usage of the server with `-t 4` if someone is interested.

![Mongodb Benchmark system resource usage](https://serpapi.com/blog/content/images/2022/09/image-1.png)

It is disappointing that MongoDB v3.4.24 leads in almost all write operation tests (insert, update and delete) and most aggregation tests. v4.4.16 wins in the rest of the aggregation tests. v5.0.12 is the biggest loser. And finally, 6.0.1 wins for all queries.

There's no reason for us to migrate to v4.4.16 and v5.0.12. As for v6.0.1, although it received an impressive query performance bump, probably related to the [Slot-Based Query Execution Engine](https://www.mongodb.com/docs/manual/release-notes/6.0/#slot-based-query-execution-engine), its write operation performance was only around 60% of v3.4.24, which was the worst among all four versions. We won't risk trading writing performance with querying.


## Driver benchmark

We've done further tests on MongoDB ruby drivers (`mongo` gem, versions v2.13.0 and v2.18.1) by inserting a certain number of documents and measuring the duration. Here's the script for testing:

```ruby
gem 'mongo', '=2.13.0'
require 'mongo'
require 'digest'
require 'benchmark'

Mongo::Logger.logger.level = Logger::WARN
client = Mongo::Client.new('mongodb://10.1.0.2:27017/test')

puts Benchmark.measure {
  100_000.times do |index|
    client[:users].insert_one(name: Digest::MD5.hexdigest(index.to_s))
  end
}
```

Results: (mongo v2.18.1 doesn't support MongoDB v3.4.24)

![MongoDB Insert Time](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6lcy2os9f6cas5ui084d.png)

We can see that mongo v2.13.0 is slightly better than v2.18.1 on insert time, which is frustrating. Also, inserting performance degrades from v3.4.24 to v6.0.1.


## Out-of-memory test on `$near`

Our final test was an OOM-Killer situation that occurred in our v3.4.24 setup. It failed once we concurrently fed 5k `$near` queries to the server.

Here is the script used for testing:

```ruby
gem 'mongo', '=2.13.0'
require 'mongo'
require 'timeout'

Mongo::Logger.logger.level = Logger::WARN
client = Mongo::Client.new('mongodb://10.1.0.2:27017/test', max_pool_size: 5000)

col = client[:locations]

5000.times.map do
  Thread.new do
    col.find({
      :gps => {
        "$near" => {
          "$geometry": { type: "Point",  coordinates: [ rand(-90.000000000...90.000000000), rand(-180.000000000...180.000000000) ] },
        }
      }
    }).to_a
  end
end.each(&:join)
```

The `location` collection was seeded with 10k entries.

Unfortunately, all four versions (v3.4.24, v4.4.16, v5.0.12, v6.0.1) still got OOM-killed within 30 seconds after running the script.

There's an [issue](https://jira.mongodb.org/browse/SERVER-22224) still unresolved, claiming that `$near` is using unbounded memory.

We've also tried to stop the script right before the server crashes. Hoping that it would stop at a high memory consumption. We wanted to see if the extra memory would be released just like with GC. But sadly, it didn't happen (we waited for 3 hours).

![MongoDB Memeory Leak](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b7sf32fp2yptg26m8i5q.png)

## Conclusion

To sum up, MongoDB had not put its focus on performance optimization, at least for write operations. Writing performance became even worse on more recent versions. The only thing encouraging is that MongoDB v6 has introduced a new query engine that boosts query performance.

It is also frustrating to find out that newer versions of `mongo` ruby driver have poorer writing performance than older versions.

The memory leak issue regarding the `$near` query has not been resolved in any of the tested versions.

These results ultimately killed our enthusiasm for upgrading MongoDB at this time. We will continue to monitor and test the performance of newer versions.