Multi-Tenancy in Django
Tags: python, django, postgres, opensource
Published: 2022-09-25 21:08:23

### Inspiration
Some months back i happen to be involved in a multi-tanent based Django project (SaaS), i began my research on multi-tanency and multi-tanancy in django. After researching for some time, i came across the [django_tenant](https://github.com/django-tenants/django-tenants) library. I studied its documentation and i was able to integrate it into the project i was working on and was able to solve my problem.

After using the library, i became curious about how the library is able to solve the problem internally, So i went further to study extensively how the library is implemented. This is what brought me to writing this article, as i feel it will be good to share how this library is able to solve the multi-tenancy problem in Django.

### Introduction
First of all what is multi-tenancy? 

Multi-tenancy is a software architecture in which a single instance of a software runs on a server and serves multiple tenants. These systems are designed in a shared manner rather than dedicated or isolated. A tenant is a group of users who share a common access with specific privileges in a multi-tenant application.

When designing multi-tenant software, there are typically three different types of architectures.

**Isolated Approach**: In this approach, each tenant has it’s own dedicated database that stores its data.

**Semi Isolated Approach**: Here the entire software uses single database, but each tenant has its own schema in which the tenant's data is stored.

**Shared Approach**: In this approach, all tenants share the same database and schema. mostly, there is a main tenant table, which all other tables are relating to.

This article focused on the second approach (Semi Isolated Approach) with PostgreSQL database, which seems to be the ideal compromise between simplicity and performance.

Before we dive into this article, let talk about schema. What is a schema in a database?

Relational databases have different concepts of schema, but in terms of PostgreSQL database, we can think of schema as a directory in an operating system, where each directory (schema) has it’s own set of files (tables and objects). This allows the same table name and objects to be used in different schemas without conflict.

_Enough of definitions! Now back to Django._

In Django, it can be quite challenging to come up with a multi tenant solution as Django currently provides no simple way to support multiple tenants applications in a single project instance. This article is here to address this issue. by extensively explaining how we can tackle multi-tenancy problem withing single Django project instance.

### How it Works
Tenants are identified via their host name (i.e tenant.domain.com). This information is stored on a table on the public schema. Whenever a request is made, the host name is used to match a tenant in the database. If there’s a match, the search path is updated to use this tenant’s schema. So from now on all queries will take place at the tenant’s schema. For example, suppose you have a tenant customer at http://customer.example.com. Any request coming from customer.example.com will automatically use customer’s schema and make the tenant available at the request. If no tenant is found, a 404 error is raised. This also means you should have a tenant for your main domain, typically using the public schema.

In order to achieve this, a lot of customization need to be done which are as follows:

- Modify the Django PostgreSQL database wrapper to enable setting of db search path to the requested tenant's schema.
- Implement a custom migration executor for multi schema db migrations.
- Implement tenant model to handle tenant and schema creation.
- Implement a custom middleware to handle request and tenant mapping.

**_Throughout this article, the public schema name is assumed to be 'public'_**

### Modifying the Django PostgreSQL database wrapper

First we're going to begin by modifying the default Django postgreSQl database introspection to enable schema based db searching.

```python
from django.db.backends.postgresql.introspection import DatabaseIntrospection

class DatabaseSchemaIntrospectionSearchPathContext:
    def __init__(self, cursor, connection):
        self.cursor = cursor
        self.connection = connection
        self.original_search_path = None

    def __enter__(self):
        self.cursor.execute('SHOW search_path')
        self.original_search_path = self.cursor.fetchone()[0].split(',')
        self.cursor.execute(f"SET search_path = '{self.connection.schema_name}'")

    def __exit__(self, *args, **kwargs):
        formatted_search_paths = ', '.join(
            f"'{search_path.strip()}'"
            for search_path in self.original_search_path
        )
        self.cursor.execute(f'SET search_path = {formatted_search_paths}')


class DatabaseSchemaIntrospection(DatabaseIntrospection):
    def get_table_list(self, cursor):
        with DatabaseSchemaIntrospectionSearchPathContext(cursor=cursor, connection=self.connection):
            return super().get_table_list(cursor)

    def get_table_description(self, cursor, table_name):
        with DatabaseSchemaIntrospectionSearchPathContext(cursor=cursor, connection=self.connection):
            return super().get_table_description(cursor, table_name)

    def get_sequences(self, cursor, table_name, table_fields=()):
        with DatabaseSchemaIntrospectionSearchPathContext(cursor=cursor, connection=self.connection):
            return super().get_sequences(cursor, table_name, table_fields)

    def get_key_columns(self, cursor, table_name):
        with DatabaseSchemaIntrospectionSearchPathContext(cursor=cursor, connection=self.connection):
            return super().get_key_columns(cursor, table_name)

    def get_constraints(self, cursor, table_name):
        with DatabaseSchemaIntrospectionSearchPathContext(cursor=cursor, connection=self.connection):
            return super().get_constraints(cursor, table_name)

```
The `DatabaseSchemaIntrospectionSearchPathContext`is a context manager that set the db search path to the requested tenant's schema, and restores the original search path of the cursor after the method called in the introspection class has been executed. We then defined `DatabaseSchemaIntrospection` class which wraps some of the Django's PostgreSQL introspection (`DatabaseIntrospection`) methods implementation with the `DatabaseSchemaIntrospectionSearchPathContext`.

Next we're going to define a class which will extend the Django's default PostgreSQL database wrapper and modify it to make use of our `DatabaseSchemaIntrospection` class.

```python
import re
import warnings
from django.conf import settings
from importlib import import_module

import DatabaseSchemaIntrospection
from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import ValidationError
import django.db.utils
import psycopg2

ORIGINAL_BACKEND = getattr(settings, 'ORIGINAL_BACKEND', 'django.db.backends.postgresql')

original_backend = import_module(ORIGINAL_BACKEND + '.base')

class DatabaseWrapper(original_backend.DatabaseWrapper):
    def __init__(self, *args, **kwargs):
        self.search_path_set_schemas = None
        self.tenant = None
        self.schema_name = None
        super().__init__(*args, **kwargs)

        self.introspection = DatabaseSchemaIntrospection(self)

        self.set_schema_to_public()

    def close(self):
        self.search_path_set_schemas = None
        super().close()

    def set_tenant(self, tenant, include_public=True):
        self.tenant = tenant
        self.schema_name = tenant.schema_name
        self.include_public_schema = include_public
        self.set_settings_schema(self.schema_name)
        self.search_path_set_schemas = None
        ContentType.objects.clear_cache()

    def set_schema(self, schema_name, include_public=True):
        self.set_tenant(FakeTenant(schema_name=schema_name), include_public)

    def set_schema_to_public(self):
        self.set_tenant(FakeTenant(schema_name='public'))

    def set_settings_schema(self, schema_name):
        self.settings_dict['SCHEMA'] = schema_name

    def _cursor(self, name=None):
        if name:
            # Only supported and required by Django 1.11 (server-side cursor)
            cursor = super()._cursor(name=name)
        else:
            cursor = super()._cursor()

        return cursor

class FakeTenant:
    """
    You can't import any db model in a backend (apparently?), so this class is used
    for wrapping schema names in a tenant-like structure.
    """
    def __init__(self, schema_name):
        self.schema_name = schema_name
```
In the `DatabaseWrapper` class above, we introduced some tenants specific variables to the default Django's `DatabaseWrapper` instance variables, and modified the wrapper's introspection to point to our custom introspection (`DatabaseSchemaIntrospection`) class instead of Django's default (`DatabaseIntrospection`), we then add custom methods which handles setting of tenants to the active db connection via Djangos' connection object (`django.db.connection`).

### Implementing a multi-schema migration executor
Here we're going to start by creating a migration runner function which we're going to be using for running schema-aware migrations.

```python
import sys
from django.db import transaction
from django.db.migrations.recorder import MigrationRecorder
from django.db import connections, DEFAULT_DB_ALIAS
from django.core.management.commands.migrate import Command
from django.core.management.base import OutputWrapper
from django.dispatch import Signal

def run_migrations(args, options, schema_name, allow_atomic=True, idx=None, count=None):
    connection = connections[options.get('database', DEFAULT_DB_ALIAS]
    connection.set_schema(schema_name)

    migration_recorder = MigrationRecorder(connection)
    migration_recorder.ensure_schema()

    stdout = OutputWrapper(sys.stdout)
    stderr = OutputWrapper(sys.stderr)
    if int(options.get('verbosity', 1)) >= 1:
        stdout.write(style.NOTICE("=== Starting migration"))
    Command(stdout=stdout, stderr=stderr).execute(*args, **options)

    try:
        transaction.commit()
        connection.close()
        connection.connection = None
    except transaction.TransactionManagementError:
        if not allow_atomic:
            raise
        pass

    connection.set_schema_to_public()
    Signal().send(run_migrations, schema_name=schema_name)
```
Above we defined `run_migrations` function which execute schema aware db migration. Notice how we're using Django's connection object to set_schema to the active db connection. This is archived as a result of the customization we did to the Django's PostgresQL `DatabaseWrapper`.

Next we're going to define our own migration executor class which will be responsible for running the db migration using the `run_migrations` function defined above.

```python
class MigrationExecutor:
    def __init__(self, args, options):
        self.args = args
        self.options = options
        self.PUBLIC_SCHEMA_NAME = 'public'

    def migrate(self, tenants=None):
        tenants = tenants or []

        if self.PUBLIC_SCHEMA_NAME in tenants:
            run_migrations(self.args, self.options, self.PUBLIC_SCHEMA_NAME)
           tenants.pop(tenants.index(self.PUBLIC_SCHEMA_NAME))
        for idx, schema_name in enumerate(tenants):
            run_migrations(self.args, self.options, schema_name, idx=idx, count=len(tenants))
```
The `MigrationExecutor` class above is going to be called whenever the `migrate` command is called replacing the default Django `MigrationExecutor` class.

Next is to create a custom schema-aware migrate command class to override the default Django migrate command class which is not schema-aware. 

```python
class MigrateSchemasCommand:
    def add_arguments(self, parser):
        parser.add_argument('app_label', nargs='?', help='App label of an application to synchronize the state.')

    def handle(self, *args, **options):
        self.PUBLIC_SCHEMA_NAME = 'public'
        executor = MigrationExecutor(self.args, self.options)
        
        # run migration on public schema
        executor.migrate(tenants=[self.PUBLIC_SCHEMA_NAME])

        # run migration on non public schemas
        tenants = Tenant.objects.only('schema_name').exclude(schema_name=self.PUBLIC_SCHEMA_NAME).values_list('schema_name', flat=True)
        executor.migrate(tenants=tenants)


Command = MigrateSchemasCommand
```
This is a basic migration command that support only one option (app_label). In this command we make the assumption that the tenant model name is `Tenant`. We will talk about the tenant model in the next section of this article.

### Implementing tenant model for tenant and schema creation
In this section we're going to implement a mixin class which will be the parent of the main tenant class. The mixin class is going to handle the schema creation whenever a new tenant is created.

```python
from django.dispatch import Signal
from django.core.management import call_command
from django.db import models, connections, DEFAULT_DB_ALIAS

class TenantMixin(models.Model):
    auto_drop_schema = False
    auto_create_schema = True

    schema_name = models.CharField(max_length=50, unique=True, db_index=True)
    domain = models.CharField(max_length=250, unique=True, db_index=True)

    class Meta:
        abstract = True

    def __str__(self):
        return self.schema_name

    def save(self, verbosity=1, *args, **kwargs):
        connection = connections[DEFAULT_DB_ALIAS]
        is_new = self._state.adding
        has_schema = hasattr(connection, 'schema_name')
        if has_schema and is_new and connection.schema_name != 'public':
            raise Exception("Can't create tenant outside the public schema. Current schema is %s." % connection.schema_name)
        elif has_schema and not is_new and connection.schema_name not in (self.schema_name, get_public_schema_name()):
            raise Exception("Can't update tenant outside it's own schema or the public schema. Current schema is %s."
                            % connection.schema_name)

        super().save(*args, **kwargs)

        if has_schema and is_new and self.auto_create_schema:
            try:
                self.create_schema(verbosity=verbosity)
            except Exception:
                # Tenant creation failed, delete what we created and 
                # re-raise the exception
                self.delete(force_drop=True)
                raise

    def _drop_schema(self, force_drop=False):
        """ Drops the schema"""
        connection = connections[get_tenant_database_alias()]
        has_schema = hasattr(connection, 'schema_name')
        if has_schema and connection.schema_name not in (self.schema_name, get_public_schema_name()):
            raise Exception("Can't delete tenant outside it's own schema or "
                            "the public schema. Current schema is %s."
                            % connection.schema_name)

        if has_schema and schema_exists(self.schema_name) and (self.auto_drop_schema or force_drop):
            cursor = connection.cursor()
            cursor.execute('DROP SCHEMA "%s" CASCADE' % self.schema_name)

    def delete(self, force_drop=False, *args, **kwargs):
        """
        Deleting a tenant will drop the tenant's schema if the attribute
        auto_drop_schema is set to True.
        """
        self._drop_schema(force_drop)
        super().delete(*args, **kwargs)

    def create_schema(self, verbosity=1):
        """
        Creates the schema for this tenant.
        """
        connection = connections[DEFAULT_DB_ALIAS]
        cursor = connection.cursor()
 
        # Check if schema exist before creating one.
        # Do nothing if schema already exist.
        if schema_exists(self.schema_name):
            return False

        # create the schema
        cursor.execute('CREATE SCHEMA "%s"' % self.schema_name)
        call_command('migrate_schemas',
                             tenant=True,
                             schema_name=self.schema_name,
                             interactive=False,
                             verbosity=verbosity)

        connection.set_schema_to_public()
```
 We're using the `schema_exists` function check if a schema exists in the db. We did not implement this function yet, it quite easy to implement all we have to do is to execute an SQL command. bellow is the `schema_exists` function.

```python
from django.db import connections, DEFAULT_DB_ALIAS

def schema_exists(schema_name, database=DEFAULT_DB_ALIAS):
    _connection = connections[database]
    cursor = _connection.cursor()
    exists = False

    # check if this schema exists in the db
    sql = 'SELECT EXISTS(SELECT 1 FROM pg_catalog.pg_namespace WHERE LOWER(nspname) = LOWER(%s))'
    cursor.execute(sql, (schema_name,))

    row = cursor.fetchone()
    if row:
        exists = row[0]

    cursor.close()
    return exists
```
With the `TenantMixin` class we can have a tenant model that will create a schema for its self in the database whenever an object of the model is created. We can define the tenant model however we want depending on the use case. Here we're going showcase a simple example of the tenant model.

```python
from django.db import models

class Tenant(TenantMixin):
    name = models.CharField(max_length=100)
    on_trial = models.BooleanField()
    created_on = models.DateField(auto_now_add=True)
```

### Implementing custom middleware to handle request and tenant mapping
This is the final stage of our Django multi-tanent solution in which we will implement a middleware that will be responsible for mapping requested hostname to a tenant if the tenant exist or raise tenant not found exception the tenant does not exist as explained in **How it Works** section.

```python
from django.db import connection
from django.http import Http404
from django.urls import set_urlconf
from django.utils.deprecation import MiddlewareMixin

class TenantMainMiddleware(MiddlewareMixin):
    TENANT_NOT_FOUND_EXCEPTION = Http404
    
    def hostname_from_request(request):
        """ Extracts hostname from request. Used for custom requests filtering.
            By default removes the request's port and common prefixes.
        """
        if request.get_host().split(':')[0].startswith("www."):
            return request.get_host().split(':')[0][4:]
        return request.get_host().split(':')[0]

    def get_tenant(self, tenant_model, hostname):
        return tenant_model.objects.get(domain=hostname)

    def process_request(self, request):
        # Connection needs first to be at the public schema, as this is where
        # the tenant metadata is stored.
        connection.set_schema_to_public()
        hostname = self.hostname_from_request(request)
        
        # Assuming tanent model name is Tenant.
        tenant_model = 'Tenant'
        try:
            tenant = self.get_tenant(tenant_model, hostname)
        except tenant_model.DoesNotExist:
            self.no_tenant_found(request, hostname)
            return

        request.tenant = tenant
        connection.set_tenant(request.tenant)
        self.setup_url_routing(request)

    def no_tenant_found(self, request, hostname):
        """ 
        raise exception if no tenant is found.
        """
        raise self.TENANT_NOT_FOUND_EXCEPTION('No tenant for hostname "%s"' % hostname)

    def setup_url_routing(request, force_public=False):
        """
        Sets the correct url conf based on the tenant
        """
        public_schema_name = 'public'
        
        if force_public or request.tenant.schema_name == 'public'):
            # assuming the configured PUBLIC_SCHEMA_URLCONF is urls_public.
            request.urlconf = 'urls_public'
```

### Conclsion
Achieving multi-tanency in a single Django project instance wasn't straightforward on its own, but with [django_tenant](https://github.com/django-tenants/django-tenants) library it became very easy to achieve. This article provides a detailed explanation on how multi-tenancy can be archived in a single Django project instance using PostgreSQL database..

### Check these out!
- [Multi-tenancy](https://en.wikipedia.org/wiki/Multitenancy)
- [Schema](www.postgresql.org/docs/9.1/static/ddl-schemas.html)
- [django_tenant documentation](https://django-tenants.readthedocs.io/en/latest)

### Social
- [Github](https://github.com/mameen7)
- [Linkedin](https://www.linkedin.com/in/ahmad-ameen-403211159/)

**I'm new to blogging, please like and share this as much as you think I deserve!**