How to migrate Firestore schema
Tags: firebase, firestore, dart, migration
Published: 2022-10-20 11:02:06

## Intro

How you store your data changes over time. Some cases, you need to modify current data to be fit in a new data model structure. If your using SQL DB, you can write a migration script and run it. But if you’re using Firebase Firestore, this task can be tricky since itself has no concept of migration. 

In order to do that, you can use Firebase Cloud Function, write some code using Firebase Admin SDK, and run it. It’s possible, but what if you’re using free plan(you can’t use Cloud Function unless you give them a payment information) or using Dart?(according to pub.dev, you can’t use official firebase package in pure dart, you must use Flutter)

I looked it up on this topic and found an awesome package called [fireway](https://github.com/kevlened/fireway). It inspired me to work on a same problem, but in this case, with Pure Dart.

In this post, I would like to share what I’ve built and how to use it.

## What I’ve used

- Dart console application
- google api related packages
- `service-account.json` for authentication
- `dart_console` for display text in a console
- `version` for dealing with migration version

## Github repository for source code

You can clone this public repository and modify freely.

- [https://github.com/swimmingkiim/firestore_migration](https://github.com/swimmingkiim/firestore_migration)

## How to use this Dart application

(Assum you already cloned the repository)

## 1. Create service account json file

First, you need service account json file. It is needed in order to automate authentication(without login all the time). You can create service account file in Google Developer Console. The link below explains how to do it.

- [https://cloud.google.com/iam/docs/creating-managing-service-account-keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)

The content is something like this.

```json
{
  "type": "service_account",
  "project_id": "PROJECT_ID",
  "private_key_id": "KEY_ID",
  "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY\n-----END PRIVATE KEY-----\n",
  "client_email": "SERVICE_ACCOUNT_EMAIL",
  "client_id": "CLIENT_ID",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT_EMAIL"
}
```

 `WARNING` → DO NOT upload this file to github publically! Anyone who has this file can lookup your project and do dirty thing. Add it to .gitignore!

You may want to move the service account file to your project root.

## 2. `flutter pub get`

Of course.

```bash
flutter pub get
```

## 3. Write your migration code

Under the `lib/migrations/versions` , you’ll find 0_0_1, 0_0_2, 0_0_3 dart files. You must follow [semver](https://semver.org/) versioning rules. But in the file name, use “_” instead of “.” 

In `0_0_1.dart` , there’s a same code.

```dart
import 'package:dart_console/dart_console.dart';
import 'package:firebaseapis/firestore/v1.dart';
import 'package:firestore_migration/config/config.dart';
import 'package:firestore_migration/src/crud_document.dart';
import 'package:firestore_migration/src/crud_field.dart';
import 'package:version/version.dart';

import '../migration.dart';

class Migration_0_0_1 implements MigrationFunc {
  @override
  Version get version => Version(0, 0, 1);

  @override
  Future<void> execute(FirestoreApi firestoreApi, Console console) async {
    // [[ YOUR_MIGRATION_CODE_HERE ]]
  }
}
```

Above code is a basic structure of a migration file.

- You implements `MigrationFunc` (in `lib/migrations/migration.dart`)
- You override `version` and `execute` function.
- `execute` function takes two parameters,
    - firestoreApi
        - from `firebaseapis` package
            - [https://pub.dev/packages/firebaseapis](https://pub.dev/packages/firebaseapis)
        - use directly with the help of official document, or use prebuilt utils which I’ll explain below
    - console
        - from `dart_console` package
        - use when you write some text(success, error etc)

In order to modify your Firestore scheme, I made util classes (`CRUDDocument` and `CRUDField`). I only created some methods that I need for now. You can freely add other method. Maybe if you think it has general use cases, then please make a PR, so that others may get your help.

Sample code I wrote is something like below.

```dart
...

final crudField = CRUDField(
      firestoreApi: firestoreApi,
      console: console,
      projectId: projectId,
    );
    final crudDocument = CRUDDocument(
      firestoreApi: firestoreApi,
      console: console,
      projectId: projectId,
    );
    final documents = await crudDocument.readDocuments('test');
    for (var document in documents) {
      await crudField.createField(
          document, 'hello', Value(stringValue: 'world'));
    }
    console.writeLine('done!');
...
```

Explain : 

- Read collection called `test`
- Create field called `hello` with string value “world” to each document under `test` collection.
- Print “done!” in console

## 4. Run it

Now you can run it. But before, It’s better for you to read basic commands and options in this application.

```
firestore_migration                                                                                                                                                                                                               

                                                                                                                                                                                                                                 
A Dart console application for firestore migration                                                                                                                                                                                

                                                                                                                                                                                                                                 

                                                                                                                                                                                                                                 
[Commands]                                                                                                                                                                                                                        

                                                                                                                                                                                                                                 
    migration       : Start migration work                                                                                                                                                                                            

                                                                                                                                                                                                                                 
[Options]                                                                                                                                                                                                                         

                                                                                                                                                                                                                                 
    --version, -v                                                                                                                                                                                                                     
                    : Specify migration file version to run. Format should be match to semver(x.y.z) Default is latest.                                                                                                                              
    --service-account, -s                                                                                                                                                                                                             
                    : Path of service account json file. (required option)                                                                                                                                                                           

                                                                                                                                                                                                                                 
[Flags]                                                                                                                                                                                                                           

                                                                                                                                                                                                                                 
    --help          : Display helper text on terminal
```

Now, You can run it.

```bash
dart run ./bin/firestore_migration.dart migrate -s service_account.json -v 0.0.1
```

And if you got this, then the job you asked for is done! Check your Firestore in Firebase console to see if data changed correctly.

```
Start : Firestore Migration...                                                                                                                                                                                                    

                                                                                                                                                                                                                                 
this is 0.0.1
done!                                                                                                                                                                                                                             

                                                                                                                                                                                                                                 
Finish : Firestore Migration!
```

## Conclusion

Note that you need to `test it on the test project` is safe. And I strongly recommend you to `backup first before make any changes` with this application. 

I hope that this repository can help with those who has this specific need.

Any new feature idea or PR is very welcomed.

Cheers!

<a href="https://www.buymeacoffee.com/swimmingkiim" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/arial-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>