CouchBase Mirror Data Records
Tags: couchbase, database, datamiror
Published: 2023-07-05 04:52:00

---
title: CouchBase Mirror Data Records
published: true
description:
tags: couchbase, database, datamiror
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
published_at: 2023-07-05 04:52 +0000
---

Hi everyone, recently, I have been worked on the couchbase with a request of data mirror from one couchbase database to another.

Basically, what I have done can be summarised as: we try to export original data and use a NodeJS script to generate a query for all the original data and finally run the query to inject all data from original data (One word: Using query to copy and paste data)

Here is the query generation script: 

```js
// This script can only be suitable for the data records which is less than 2000;

const fs = require("fs");

function convert(collectionName, sourceFile, query, resultFile) {
  fs.readFile(sourceFile, "utf8", (err, data) => {
    if (err) {
      console.log("Source file read failed:", err);
      return;
    }

    const json = JSON.parse(data);

    const response = json.reduce((acc, cur, index) => {
      const key = cur[collectionName].id.toString();
      const value = cur[collectionName];
      const valueString = JSON.stringify(value);

      cur = ('("' + key + '", ' + valueString + ")").replace(/\\/g, "");

      comma = index === json.length - 1 ? "" : ", ";

      acc += cur + comma;

      return acc;
    }, "");

    if (response) {
      const final = `${query} ${response}`;
      fs.writeFile(resultFile, final, function (err) {
        if (err) return console.log(err);
        console.log(`Generated data have been saved into ${resultFile}`);
      });
    }
  });
}
```

Here is the example of how to do it:


Step 1: we need to go to the original data source we want to copy from (The `couchbase bucket` -> `collection` -> `document`).


Just provide an example here, if I want to mirror data for test-bucket (bucket name) -> test-collection (collection name) -> test-documents (document name)


Step 2: we click the Query tab on the left side menu bar

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cc4x1t2v5bsn8hwn9q6c.jpg)


Step 3: we run a query: SELECT * FROM `test-bucket`.`test-collection`.`test-documents`


Step 4: we select `EXPORT` button on the top right corner and you will see a JSON file export modal window:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/anvmyufwsd2hpxwvslcd.jpg)


Step 5: we name the file name as test-documents

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hwi44igm8skaenu6mrai.png)

Then Click Save button to export data as JSON file


Step 6: You can open the `convert.js` script I have attached below and you will see this setup from line 39 to line 45 below:

```js
convert(
  "test-documents", // the collection name
  "./test-documents.json", // the file you have exported from CouchBase (Step 5)
  "INSERT INTO `test-bucket`.`test-collection`.`test-documents`(KEY, VALUE) VALUES ", // the target bucket -> collection -> document 
  "test-documents-query.txt" // the final generated result query for data mirror / injection usage 

);
```
 

Step 7: Modify the script provided in above (Step 6) based on your specific requirements.

As an example here, we want to copy the one couchbase database test-documents (document name) data to another couchbase database test-documents (document name), we can run above script for generating the data injection /mirror query.

run 
```js
node convert.js
``` 
command in your terminal within the current folder path and you will be able to generate the new query text file called as `test-documents-query.txt`.

 
Step 8: We go to the destination / target document (eg: test-documents) and go to the Query tab (Same as Step 2) and paste the new generated query inside the `Query Editor`


Step 9 [Optional]: If you just created a new document for UK couchBase, you need to run this query first to avoid some index errors which will be reported by couchbase

```sql
CREATE PRIMARY INDEX ON `default`:`test-bucket`.`test-collection`.`test-documents`
```

*** If you already have document primary index setup already, you can skip Step 9. ***


Step 10: You will open the final generated data injection text file: `test-documents-query.txt` and copy all the contents and paste into the Query Editor

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gkog6qdi0y5ll6buqnn0.png) 

Step 11: Click the `Execute` button and starting injecting the data from one test-documents to another 🚀🚀

 
Some Additional Tips:

* Before mirror the data, better to try to use the script locally (`localhost:3000`), so you will be able to get familiar with and understand this script before actually use it 

* Always ensure the bucket name, collection name and document name are entered correctly. Otherwise, you might inject the data in a wrong document

Finally, as a reference,

You might be able to find the work I have been working on:
https://github.com/DamengRandom/n1ql-query-generator


Thank you very much for reading, hopefully it could help a little with your couchbase data mirror process.