How to migrate a Mongo Database with Ansible Playbook
Tags: mongodb, atlas, digitalocean, ansible
Published: 2023-10-31 03:15:19

Recently, a client tasked me with migrating MongoDB from [MongoDB Atlas](https://www.mongodb.com/atlas/database) to [Digital Ocean managed MongoDB service](https://www.digitalocean.com/products/managed-databases-mongodb). He had 2 reasons, one was to manage everything in Digital Ocean since the application server was in Digital Ocean, and two was to minimize the network latency, the client wanted to have the MongoDB in the same VPC as the application server.

## Prerequisites
- [Database Tools](https://www.mongodb.com/docs/database-tools/installation/installation/) (The package includes mongodump and mongorestore)
- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html)

## Migration Concept

To migrate the mongo database, we will use the backup (*mongodump*) and restore (*mongorestore*) utilities. In my case, I ran both commands from my application server in Digital Ocean. We will then use the Ansible Playbook to create a template to run it as many times as we need. In my case, the client wanted to run it twice. First, he wanted me to migrate the DB to a staging environment, and once everything worked correctly, he asked me to migrate the DB to a production environment. Ansible Playbook also provides time calculation, giving you an idea of how long the live Mongodb migration will take.

## Backup

We will back up the MongoDB from MongoDB Atlas using the *mongodump* utility.

### Syntax and Example

To backup your existing database, the mongodump syntax will be like this:
```sh
mongodump --username DB_USERNAME --out=TARGET_FOLDER --config=mongodump.cfg
```
Note that:
- **username** is the username we use to authenticate to our mongo DB.
- **out** is the target folder to store our mongo DB backup.
- **config** is the config file that will be used by mongodump.

The mongodump.cfg contains these configurations:
```sh
password: PASSWORD
uri: mongodb+srv://HOSTNAME.mongodb.net/DATABASE_NAME?retryWrites=true&w=majority
```
Change PASSWORD, HOSTNAME, and DATABASE_NAME accordingly. *retryWrites=true&w=majority* are connection parameters used by mongo DB Atlas, so leave them as they are.

## Restore
We will restore the MongoDB to Digital Ocean managed mongo DB service using the *mongorestore* utility.

### Syntax and Example

To restore to a database, the mongorestore syntax will be like this:
```sh
mongorestore --ssl --tlsInsecure --username USERNAME --config=mongorestore.cfg --nsInclude=DATABASE_NAME TARGET_FOLDER/DATABASE_NAME/
```
Note that:
- **ssl** means we will connect to mongo DB via TLS/SSL.
- **tlsInsecure** will disable the certificate validations.
- **config** is the config file that will be used by mongorestore.
- **nsInclude** specifies the namespace pattern. Since we will restore the whole database, we will use DATABASE_NAME (change this with your database name).

The mongorestore.cfg contains these configurations:
```sh
uri: mongodb+srv://HOSTNAME.mongo.ondigitalocean.com/DATABASE_NAME?authSource=admin
password: PASSWORD
```
Change PASSWORD, HOSTNAME, and DATABASE_NAME accordingly.
*authSource=admin* is a connection parameter used by Digital Ocean managed mongo DB service, so leave it as it is.

## Ansible Playbook
We will now use Ansible Playbook. First create a file named `playbook.yml`, and then put these in the file:
```yml
# How to run: ansible-playbook playbook.yaml
-
  name: Migrate mongodb
  hosts: localhost
  tasks:
    - name: Remove backup folder if exists
      ansible.builtin.file:
        path: /root/mongorestore/backup
        state: absent
    - name: Mongodump from mongodb atlas
      shell: mongodump --username USERNAME --out=backup/mongodb --config=mongodump.cfg
    - name: Mongorestore to Digital Ocean managed database - mongodb
      shell: mongorestore --ssl --tlsInsecure --username USERNAME --config=mongorestore.cfg --nsInclude=DATABASE_NAME backup/mongodb/DATABASE_NAME/
```
Change USERNAME and DATABASE_NAME accordingly.
The ansible playbook contains 3 tasks, and they are:
1. The first task will remove the backup folder if it exists. Since we will run the ansible playbook many times, we need to do this to ensure there is no trace of the previous backup
2. The 2nd task will dump mongo DB
3. The last task will restore mongo DB

Next, we need to create a file named `mongodump.cfg`, and then put these in the file:
```sh
password: PASSWORD
uri: mongodb+srv://HOSTNAME.mongodb.net/DATABASE_NAME?retryWrites=true&w=majority
```
Remember to change PASSWORD, HOSTNAME, and DATABASE_NAME accordingly.

Then create a file named `mongorestore.cfg`, and then put these in the file:
```sh
uri: mongodb+srv://HOSTNAME.mongo.ondigitalocean.com/DATABASE_NAME?authSource=admin
password: PASSWORD
```

Lastly, we will configure Ansible to display the time calculation to know how much time we need to run all the 3 tasks. According to the doc [here](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#the-configuration-file), there are 4 ways that we can configure (pick one), and they are:
- The `ANSIBLE_CONFIG` environment variable
- Create a file named `ansible.cfg` and put it in the same folder as the playbook.yml resides
- `~/.ansible.cfg`, if you are logged in as `root`, the file will be stored in `/root`. If you are logged in another user, e.g. `ubuntu`, it will be stored in `/home/ubuntu`
- Modify the file in `/etc/ansible/ansible.cfg`

Let's say we create a file named `ansible.cfg`. Once created, put these in the file:
```sh
[defaults]
callback_whitelist = profile_tasks, profile_roles, timer
```
We use `callback_whitelist` to allow some plugins to run during the playbook execution. For ansible version older than 2.0, refer to [here](https://github.com/jlafon/ansible-profile#usage-in-ansible-1x) to find out how to display the time calculation. [profile_tasks](https://docs.ansible.com/ansible/2.9/plugins/callback/profile_tasks.html), [profile_roles](https://docs.ansible.com/ansible/2.9/plugins/callback/profile_roles.html), and [timer](https://docs.ansible.com/ansible/2.9/plugins/callback/timer.html) will display a detailed information of the executed tasks.

Before we run the ansible playbook, we need to create a new terminal session so that when our SSH connection is accidentally closed, the command will continue to run in the background. Run this command:
```sh
screen -S mongodb_migration
```

To connect to the terminated terminal session, run this command:
```
screen -R
```

Once everything is fully setup, run the ansible playbook with this command:
```sh
ansible-playbook playbook.yml
```

The output will look like this:
```sh
root@docker-ubuntu:~/mongodb-migration# ansible-playbook playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Migrate mongodb] ****************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************
Tuesday 31 October 2023  03:00:31 +0000 (0:00:00.042)       0:00:00.042 *******
Tuesday 31 October 2023  03:00:31 +0000 (0:00:00.041)       0:00:00.041 *******
ok: [localhost]

TASK [Remove backup folder if exists] *************************************************************************************************************************************
Tuesday 31 October 2023  03:00:32 +0000 (0:00:01.770)       0:00:01.812 *******
Tuesday 31 October 2023  03:00:32 +0000 (0:00:01.770)       0:00:01.812 *******
changed: [localhost]

TASK [Mongodump from mongodb atlas] ***************************************************************************************************************************************
Tuesday 31 October 2023  03:00:33 +0000 (0:00:00.608)       0:00:02.420 *******
Tuesday 31 October 2023  03:00:33 +0000 (0:00:00.607)       0:00:02.420 *******
changed: [localhost]

TASK [Mongorestore to Digital Ocean managed database - mongodb] ***********************************************************************************************************
Tuesday 31 October 2023  03:04:34 +0000 (0:04:01.372)       0:04:03.793 *******
Tuesday 31 October 2023  03:04:34 +0000 (0:04:01.373)       0:04:03.793 *******
changed: [localhost]

PLAY RECAP ****************************************************************************************************************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Tuesday 31 October 2023  03:09:43 +0000 (0:05:08.507)       0:09:12.300 *******
===============================================================================
shell ----------------------------------------------------------------- 549.88s
gather_facts ------------------------------------------------------------ 1.77s
ansible.builtin.file ---------------------------------------------------- 0.61s
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
total ----------------------------------------------------------------- 552.26s
Tuesday 31 October 2023  03:09:43 +0000 (0:05:08.508)       0:09:12.301 *******
===============================================================================
Mongorestore to Digital Ocean managed database - mongodb --------------------------------------------------------------------------------------------------------- 308.51s
Mongodump from mongodb atlas ------------------------------------------------------------------------------------------------------------------------------------- 241.37s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------- 1.77s
Remove backup folder if exists ------------------------------------------------------------------------------------------------------------------------------------- 0.61s
```

I have uploaded the ansible files to a repository [here](https://github.com/budiantoip-it/mongodb-migration).

Feel free to connect with me on LinkedIn:
https://www.linkedin.com/in/budiantoip/