Improve PouchDB's initial sync with CouchDB 3x faster
Tags: couchdb, pouchdb, javascript, performance
Published: 2020-08-04 07:53:40

## Update (2020/08/07)

It is not a correct workaround to improve replications.
After having some research, it turned out that the actual bottleneck was my network topology on EC2.
My cluster nodes were deployed across different regions in order to make them disaster tolerant.
That caused the network bottleneck between nodes where the RTT was around 166ms.
I moved the nodes to the single region but in different availability zones where the RTT is less than 1ms.
Now it works very fast!
You don't need to hack PouchDB but check your network performance.

## Problem

I was not happy with the replication performance between PouchDB and CouchDB, especially when performing the initial sync.
I'm building a note-taking app called [Inkdrop](https://www.inkdrop.app/) which supports data sync between devices using a clustered CouchDB. I found that a user complains about the slow sync speed. I was aware of that, so I decided to work on it.
There is an issue on the PouchDB's GitHub repository:

* [CouchDB - PouchDB initial replication is very slow · Issue #7634 · pouchdb/pouchdb](https://github.com/pouchdb/pouchdb/issues/7634)

But it's inactive and seems that nobody has solved the issue at the moment.

So, I tried to find the actual bottleneck in the replication process. It turned out that `bulkGet` (CouchDB: `_bulk_get`) takes very long time. It took 6 secs for fetching 100 documents. It's way slower than `allDocs` (CouchDB: `_all_docs`). I suspected it was causing the problem.
I found that PouchDB is specifying `revs: true` and `latest: true` parameters when calling `bulkGet` API:

* [pouchdb/getDocs.js at master · pouchdb/pouchdb](https://github.com/pouchdb/pouchdb/blob/master/packages/node_modules/pouchdb-replication/src/getDocs.js#L58)

It turned out that the response time becomes significantly faster when calling it without those params but with `r: 1` option added.
The `r=1` parameter means you read data from 1 replica node.
It lets the cluster avoid reading data from multiple nodes.
Getting revisions by specifying `revs: true` or `latest: true` is slow because the database has to check the document history. But after having looked into the source code of PouchDB, it seems to be not using `_revisions` field in the replication process if I were correct. The `latest` param is to avoid race conditions where another client updated the doc while syncing. But my app uses "one database per user"  pattern, so I guess the race condition problem would be rare.

In conclusion, I accomplished improving the sync speed 3x faster by removing `revs` and `latest` params and added `r: 1` to the `bulkGet` internal calls in PouchDB with the hack with the core modules as following.

In [pouchdb-replication/src/getDocs.js#L46](https://github.com/pouchdb/pouchdb/blob/master/packages/node_modules/pouchdb-replication/src/getDocs.js#L46):

```js
function createBulkGetOpts(diffs) {
  var requests = [];
  Object.keys(diffs).forEach(function (id) {
    var missingRevs = diffs[id].missing;
    missingRevs.forEach(function (missingRev) {
      requests.push({
        id: id,
        rev: missingRev
      });
    });
  });

  return {
    docs: requests,
    /* DELETE
    revs: true,
    latest: true
    */
  };
}
```

In [pouchdb-adapter-http/src/index.js#L341](https://github.com/pouchdb/pouchdb/blob/master/packages/node_modules/pouchdb-adapter-http/src/index.js#L341):

```js
  api.bulkGet = coreAdapterFun('bulkGet', function (opts, callback) {
    var self = this;

    function doBulkGet(cb) {
      var params = {};
      if (opts.revs) {
        params.revs = true;
      }
      if (opts.attachments) {
        /* istanbul ignore next */
        params.attachments = true;
      }
      if (opts.latest) {
        params.latest = true;
      }
      // ADD THIS
      params.r = 1
      fetchJSON(genDBUrl(host, '_bulk_get' + paramsToStr(params)), {
        method: 'POST',
        body: JSON.stringify({ docs: opts.docs})
      }).then(function (result) {
        if (opts.attachments && opts.binary) {
          result.data.results.forEach(function (res) {
            res.docs.forEach(readAttachmentsAsBlobOrBuffer);
          });
        }
        cb(null, result.data);
      }).catch(cb);
    }

```

Now it takes only around 2 seconds for 100 docs which is 3x faster than before.
It works fine with my app for now.