Django: Migrate --fake-initial gotchas
Tags: python, django, migration, db
Published: 2018-09-06 01:36:45

---
title: Django: Migrate --fake-initial gotchas
published: true
description: Migrate --fake-initial usually used to start using migrations framework on existing database where it will detect if table already exists, it will skip creating the table and mark the migrations as applied. But it detect table in group, so one table is missing, it assume all the tables within the migration as not exists.
tags: python, django, migration, db
---

Some gotcha we got into while upgrading an old system build since Django-1.0, which then gradually upgraded to 1.4, 1.8 and finally now to 1.11 (the last 1.x version). For notes, database migrations framework only landed in Django-1.7. In this new migrations framework, you have to explicitly generate a migration files. Prior to that, the `syncdb` command will generate tables (which doesn't exists) on the fly based on your model definition. Subsequent changes to the table, like adding new column has to be handled manually through external means.

Up until 1.8, having explicit migrations files not yet mandatory so the `migrate` command basically exhibit same behavior as the old `syncdb`. But since we're updating to 1.11, we have to create the migrations files for all the apps in the system. This initial migrations basically involved creating the tables when we run `migrate` command and since all the migrations not yet tracked, it will try to apply it which obviously will fail because all the tables already exists.

For that reason, django provide 2 options:-

```
--fake¶
Tells Django to mark the migrations as having been applied or unapplied, but
without actually running the SQL to change your database schema.

This is intended for advanced users to manipulate the current migration state
directly if they’re manually applying changes; be warned that using --fake runs the
risk of putting the migration state table into a state where manual recovery will
be needed to make migrations run correctly.

--fake-initial¶
Allows Django to skip an app’s initial migration if all database tables with the
names of all models created by all CreateModel operations in that migration already
exist. This option is intended for use when first running migrations against a
database that preexisted the use of migrations. This option does not, however,
check for matching database schema beyond matching table names and so is only safe
to use if you are confident that your existing schema matches what is recorded in
your initial migration.
```

We decided to use `--fake-initial` for the the `migrate`. But we got error table already exists. It doesn't make sense at first, as we can read above it will detect if table already exists and then just "fake apply" the migrations. And we know for sure the table really exists, so why Django still want to create that table ?

I stepped through the migrations code in `django/db/migrations/executor.py` and put a `pdb.set_trace()` in this part:-

```
        # Make sure all create model and add field operations are done
        for operation in migration.operations:
            import pdb;pdb.set_trace()
            if isinstance(operation, migrations.CreateModel):
                model = apps.get_model(migration.app_label, operation.name)
                if model._meta.swapped:
```

Stepping through this and fortunately the error happened pretty early, I can see it correctly detected the table exists:-

```
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(337)detect_soft_applied()
-> if should_skip_detecting_model(migration, model):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(339)detect_soft_applied()
-> if model._meta.db_table not in existing_table_names:
(Pdb) l
334                         # We have to fetch the model to test with from the
335                         # main app cache, as it's not a direct dependency.
336                         model = global_apps.get_model(model._meta.swapped)
337                     if should_skip_detecting_model(migration, model):
338                         continue
339  ->                 if model._meta.db_table not in existing_table_names:
340                         return False, project_state
341                     found_create_model_migration = True
342                 elif isinstance(operation, migrations.AddField):
343                     model = apps.get_model(migration.app_label, operation.model_name)
344                     if model._meta.swapped:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(341)detect_soft_applied()
-> found_create_model_migration = True
```

Until it came to this iteration:-

```
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(330)detect_soft_applied()
-> import pdb;pdb.set_trace()
(Pdb) operation.name
'Image'
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(331)detect_soft_applied()
-> if isinstance(operation, migrations.CreateModel):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(332)detect_soft_applied()
-> model = apps.get_model(migration.app_label, operation.name)
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(333)detect_soft_applied()
-> if model._meta.swapped:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(337)detect_soft_applied()
-> if should_skip_detecting_model(migration, model):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(339)detect_soft_applied()
-> if model._meta.db_table not in existing_table_names:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(340)detect_soft_applied()
-> return False, project_state
```

So why it detected the table as not exists ? Inspecting the variable `existing_table_names` finally gave me a light. The table indeed does not exists !

```
(Pdb) model._meta.db_table not in existing_table_names
True
(Pdb) model._meta.db_table
'blog_image'
(Pdb) existing_table_names
['auth_group', 'auth_group_permissions', 'auth_permission', 'auth_user', 'auth_user_groups', 'auth_user_user_permissions', 'blog_article', 'django_admin_log', 'django_content_type', 'django_migrations', 'django_session']
```

So what happened was that it detecting the table as a group, so if there's one table does not exists, then the whole of that particular migrations will be applied, which mean also creating table that already exists. In our case, this happened because when dumping the production database to test this migrations, we exclude some tables with large data to keep the dump small. Which mean the resulting structure not really matching what described in the migrations file, which already being warned in the documentation above, sigh.