This new mongoose feature can improve your MongoDB writes up to 2.6x.
Tags: mongodb, javascript, performance, codequality
Published: 2021-11-18 18:35:07

## Almost always, Backend == Database
It's no secret that one of the most important aspects of a typical backend application is to serve as a nice layer between a user-facing application and the database.

This means that mastering your database and optimizing its performance can make the difference between a backend app that can handle hundreds of thousands of users simultaneously, and an application that breaks with the first few hundred real users.

## Bulk operations for better performance
One popular and easy-to-implement technique that can have significant performance gains is reducing the number of roundtrips between a backend application and a database or any external service by grouping operations together.

So for example if you want to fetch 100 orders from the database, you could:
* Create 100 different single queries, resulting in 100 network roundtrips between your backend and the database.
* Create 1 query that fetches all 100 users at once, resulting in a single network roundtrip.

Even though the size of the data is the same, fewer network roundtrips will be significantly faster; Other than the number of network roundtrips, grouping operations allows the database to optimize internally, one example of an optimization is reducing the number of index lookups.

## Mongoose with `Model.bulkSave(...)`
In mongoose v5.13.0 we added a feature that leverages this technique by sending multiple writes in bulk instead of sending each operation individually: **[Model.bulkSave(...)](https://mongoosejs.com/docs/api.html#model_Model.bulkSave)** which utilizes the native MongoDB **[bulkWrite(...)](https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/)** under the hood.
The good news is that the API is intentionally designed to require minimal changes to your existing code to gain those performance benefits.

If you have a scenario where you have 100 users, and for each user, you'd like to find their cart, and if they don't have a cart in the database, you'd like to create a new one, you could do that by one of two ways

*Variant A)* The following code snippet sends one `updateOne/insertOne` command to the database for each cart:
```js
const users = await User.find({ cityId: 1 });
await Promise.all(users.map(async function (user){
  let cart = await  Cart.findOne({ userId: user._id });
  if (!cart) {
    cart = new Cart({ userId:user._id });
  }

  cart.itemsIds.addToSet(item._id);
  await cart.save();
}));
```

*Variant B)* Here's how the code snippet above can be converted to use `bulkSave(...)`, resulting in a single database command, regardless of how many documents we're trying to save:
```js
const users = await User.find({ cityId: 1 });
const carts = await Promise.all(users.map(async function (user){
  let cart = await  Cart.findOne({ userId: user._id });
  if (!cart) {
    cart = new Cart({ userId:user._id });
  }

  cart.itemsIds.addToSet(item._id);
  // instead of saving each cart individually
  // we'll return them now and save them all in bulk later
  return cart;
}));


// calls Cart.bulkWrite under the hood
const writeResult = await Cart.bulkSave(carts); 
```
The first example sends N operations to the database, creating a heavy load on the wire, while the second one sends a single operation.


## Performance comparison:
[`Model.bulkSave(...)`](https://mongoosejs.com/docs/api.html#model_Model.bulkSave) can be ~3x faster than regular [`Document#save(...)`](https://mongoosejs.com/docs/documents.html#updating-using-save), to test this yourself, run the code from this [gist](https://gist.github.com/AbdelrahmanHafez/c6780daee01600844944f10765cd0b80).


![Model.bulkSave(...) vs Document#save(...) performance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7ivodnszmevzua3mu6t4.png)


## When to use `bulkSave`, and when to use `save`?
So, should you always use `bulkSave` over `save`?
MongoDB has [a limit](https://docs.mongodb.com/manual/reference/limits/#mongodb-limit-Write-Command-Batch-Limit-Size) to the size of data you can send with bulkWrite which is 100,000 operations per batch. I would [chunk](https://lodash.com/docs/4.17.15#chunk) my documents to be sent 10k at a time just to be safe, and still get the significant performance benefits of bulkSave.

If you're usually dealing with less than 10k documents, you should be safe always using `bulkSave`, otherwise chunk your documents to be processed 10k at a time.


How do you feel about `bulkSave`?
Would you use it in your application?
What other performance techniques that you have found useful over the time?

Let me know in the comments!