Model Field - Django ORM Working - Part 2
Tags: python, django, database
Published: 2021-06-27 08:27:12

The last post covered [the structure of Django Model](https://dev.to/kracekumar/structure-django-orm-working-part-1-10ch). This post covers how the model field works, what are the some important methods and functionality and properties of the field.

[Object-Relational Mapper](https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping) is a technique of declaring, querying the database tables using Object relationship in the programming language. Here is a sample model declaration in Django.


```python

class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')
```

Each class inherits from `models.Model` becomes a table inside the SQL database unless explicitly marked as abstract. The `Question` model becomes `<app_name>_question` table in the database. `question_text` and `pub_date` become columns in the table. The properties of the each field are declared by instantiating the respective class. Below is the method resolution order for `CharField`.

```python
In [5]: models.CharField.mro()
Out[5]:
[django.db.models.fields.CharField,
 django.db.models.fields.Field,
 django.db.models.query_utils.RegisterLookupMixin,
 object]
```



`CharField` inherits `Field` and `Field` inherits `RegisterLookUpMixin`.



## High-level role of Field class



1. The role of field class is to map type of the field to SQL database type.

2. Serialization - to convert the Python object into relevant SQL database value.
3. DeSerialization - to convert the SQL database value into Python object.
4. Check declared validations at the field level and built-in checks before serializing the data. For example, in a `PositiveIntegerField` the value should be greater than zero - built-in constraint.



## Structure of Field Class



```python
# Find out all the classes inheriting the Field

In [7]: models.Field.__subclasses__()
Out[7]:
[django.db.models.fields.BooleanField,
 django.db.models.fields.CharField,
 django.db.models.fields.DateField,
 django.db.models.fields.DecimalField,
 django.db.models.fields.DurationField,
 django.db.models.fields.FilePathField,
 django.db.models.fields.FloatField,
 django.db.models.fields.IntegerField,
 django.db.models.fields.IPAddressField,
 django.db.models.fields.GenericIPAddressField,
 django.db.models.fields.TextField,
 django.db.models.fields.TimeField,
 django.db.models.fields.BinaryField,
 django.db.models.fields.UUIDField,
 django.db.models.fields.json.JSONField,
 django.db.models.fields.files.FileField,
 django.db.models.fields.related.RelatedField,
 django.contrib.postgres.search.SearchVectorField,
 django.contrib.postgres.search.SearchQueryField,
 fernet_fields.fields.EncryptedField,
 enumchoicefield.fields.EnumChoiceField,
 django.contrib.postgres.fields.array.ArrayField,
 django.contrib.postgres.fields.hstore.HStoreField,
 django.contrib.postgres.fields.ranges.RangeField]
```



Here `fernet_fields` is a third-party library which implements the `EncryptedField` by inheriting the `Field` class. Also these are high level fields. For example, Django implements other high-level fields which inherit the above fields.

For example, `EmailField` inherits the `CharField`.

```python
In [10]: models.CharField.__subclasses__()
Out[10]:
[django.db.models.fields.CommaSeparatedIntegerField,
 django.db.models.fields.EmailField,
 django.db.models.fields.SlugField,
 django.db.models.fields.URLField,
 django.contrib.postgres.fields.citext.CICharField,
 django_extensions.db.fields.RandomCharField,
 django_extensions.db.fields.ShortUUIDField]
```

Here is the `Field` class initializer signature


```python
In [11]: models.Field?
Init signature:
models.Field(
    verbose_name=None,
    name=None,
    primary_key=False,
    max_length=None,
    unique=False,
    blank=False,
    null=False,
    db_index=False,
    rel=None,
    default=<class 'django.db.models.fields.NOT_PROVIDED'>,
    editable=True,
    serialize=True,
    unique_for_date=None,
    unique_for_month=None,
    unique_for_year=None,
    choices=None,
    help_text='',
    db_column=None,
    db_tablespace=None,
    auto_created=False,
    validators=(),
    error_messages=None,
)
```



The `Field` initializer contains 22 arguments. Most of the arguments are related to SQL database column properties and rest of the arguments are for Django admin and model forms.

For example, Django provides Admin interface to browse the database records and allows you to edit. `blank` parameter determines whether the field is required while filling up data in the admin interface and custom form. `help_text` field is used while display the form.

The most commonly used fields are `max_length, unique, blank, null, db_index, validators, default, auto_created`. `null` attribute is a boolean type when set to `True`, the allows null value while saving to the database. `db_index=True` created a `B-Tree` index on the column.  `default` attribute stores the default value passed on to the database, when the value for the field is missing.

`validators` attribute contains list of validators passed on by the user and Django's internal validators. The function of the validator is to determine the value is valid or not.  For example, in our `question_text` field declaration `max_length` is set to `200`. When the field value is greater than 200, Django raises `ValidationError`.  `max_length` attribute is useful only for text field and `MaxLengthValidator` will be missing in non-text fields.

```python
In [29]: from django.core.exceptions import ValidationError

In [30]: def allow_odd_validator(value):
    ...:     if value % 2 == 0:
    ...:         raise ValidationError(f'{value} is not odd number')
    ...:

In [31]: int_field = models.IntegerField(validators=[allow_odd_validator])

In [32]: int_field.validators
Out[32]:
[<function __main__.allow_odd_validator(value)>,
 <django.core.validators.MinValueValidator at 0x1305fdac0>,
 <django.core.validators.MaxValueValidator at 0x1305fda30>]

In [33]: # let's look into question_text field validators

In [38]: question_text.validators
Out[38]: [<django.core.validators.MaxLengthValidator at 0x12e767fa0>]
```



As long as the validator function or custom class doesn't raise exception, the value is considered as `valid`.

The details of each field can found in the [Django model field reference](https://docs.djangoproject.com/en/3.2/ref/models/fields/)

## Field Methods



```python
In [41]: import inspect

In [44]: len(inspect.getmembers(models.Field, predicate=inspect.isfunction))
Out[44]: 59

In [45]: len(inspect.getmembers(models.Field, predicate=inspect.ismethod))
Out[45]: 6
```



The `Field` class consists of (along with inherited ones) 65 methods. Let's look at some of the important ones.

### to_python

`to_python` method is responsible to convert the value passed on to the model  during intialization. For example, `to_python` for `IntegerField` will convert the value to Python integer. The original value could be `string` or `float`. Every field will override `to_python` method. Here is an example of `to_python` method invocation on an `IntegerField`.

```python
In [46]: int_field.to_python
Out[46]: <bound method IntegerField.to_python of <django.db.models.fields.IntegerField>>

In [47]: int_field.to_python('23')
Out[47]: 23

In [48]: int_field.to_python(23)
Out[48]: 23

In [49]: int_field.to_python(23.56)
Out[49]: 23
```



### get_db_prep_value

`get_db_prep_value` method is responsible to convert Python value to SQL database specific value. Each field may have a different implementation depending on field type. For example, `Postgres` has a native `UUID` type, whereas in `SQLite` and `MySQL` Django uses `varchar(32)`. Here is the implementation for `get_db_prep_value` from `UUIDField`.

```python
def get_db_prep_value(self, value, connection, prepared=False):
	if value is None:
  	return None
  if not isinstance(value, uuid.UUID):
    value = self.to_python(value)

  if connection.features.has_native_uuid_field:
  	return value
  return value.hex
```

 `connection` is a Database Connection or Wrapper object of underlying database. Below is an example output from a `Postgres Connection` and `SQLite Connection` for uuid field check.

```python
In [50]: from django.db import connection
    ...:

In [51]: connection
Out[51]: <django.utils.connection.ConnectionProxy at 0x10e3c8970>

In [52]: connection.features
Out[52]: <django.db.backends.postgresql.features.DatabaseFeatures at 0x1236a6a00>

In [53]: connection.features.has_native_uuid_field
Out[53]: True
```

```python
In [1]: from django.db import connection

In [2]: connection
Out[2]: <django.utils.connection.ConnectionProxy at 0x10fe3b4f0>

In [3]: connection.features
Out[3]: <django.db.backends.sqlite3.features.DatabaseFeatures at 0x110ba5d90>

In [4]: connection.features.has_native_uuid_field
Out[4]: False
```



One thing to note, Django uses `psycopg2` driver for Postgres and it will take care of handling UUID specific to Postgres because UUID Python object needs to be converted to string or bytes before sending to the Postgres server.

Similar to `get_db_prep_value`, `get_prep_value`  which converts Python value to `query value`.

### formfield

Django supports `ModelForm` which is one to one mapping of HTML form to Django model. The Django admin uses `ModelForm` . The form consists of several fields. Each field in the form maps to field in the model. So Django can automatically construct the form with a list of validators from the model field.

Here is the implementation for the UUIDField.

```python
def formfield(self, **kwargs):
	return super().formfield(**{
  	'form_class': forms.UUIDField,
    **kwargs,
   })
```

When you create a custom database field, you need to create a custom form field to work with Django admin and pass it as an argument to super class method.

### deconstruct

`deconstruct` method returns value for creating an exact copy of the field. The method returns a tuple with 4 values.

- The first value is the `name` of the field passed during initialisation. The default value is `None`.
- The import path of the field.
- The list of positonal arguments passed during the field creation.
- The dictionary of keyword arguments passed during the field creation.

```python
In [62]: # Let's see the question_text deconstruct method return value

In [63]: question_text.deconstruct()
Out[63]: (None, 'django.db.models.CharField', 
[], {'max_length': 200})

In [65]: # let's create a new integer field with a name

In [66]: int_field = models.IntegerField(name='int_field', validators=[allow_odd_validator])

In [67]: int_field.deconstruct()
Out[67]:
('int_field',
 'django.db.models.IntegerField',
 [],
 {'validators': [<function __main__.allow_odd_validator(value)>]})

In [68]: models.IntegerField(**int_field.deconstruct()[-1])
Out[68]: <django.db.models.fields.IntegerField>

In [69]: int_2_field = models.IntegerField(default=2)

In [70]: int_2_field.deconstruct()
Out[70]: (None, 'django.db.models.IntegerField', 
[], {'default': 2})

```

Also when you implement a custom field, you can override the deconstruct method. Here is the `deconstruct` implementation for `UUIDField`.

```python
def deconstruct(self):
    name, path, args, kwargs = super().deconstruct()
    del kwargs['max_length']
    return name, path, args, kwargs
```



### __init__

`__init__` method is a good place to override some of the default values. For example, `UUIDField` max_length should always be `32` irrespective of the value passed on. In the decimal field, `max_digits` can be modified during initialization.

Here is the `UUIDField` initializer method implementation.

```python
def __init__(self, verbose_name=None, **kwargs):
    kwargs['max_length'] = 32
    super().__init__(verbose_name, **kwargs)
```

### db_type

`db_type` method takes Django connection as an argument and returns the database specific implementation type for this field. The method takes connection as an argument. Here is the output of db_type for Postgres and SQLite.

```python
In [72]: # Postgres

In [73]: uuid_field = models.UUIDField()

In [74]: uuid_field.db_type(connection)
Out[74]: 'uuid'
```



```python
In [8]: # Sqlite

In [9]: uuid_field = models.UUIDField()

In [10]: uuid_field.rel_db_type(connection)
Out[10]: 'char(32)'
```

`get_internal_type` method returns `internal` Python type which is companion to the `db_type` method. In practice, Django fields type and database field mapping is maintained as class variable in `DatabaseWrapper`. You can find, [Django fields and Postgres fields mapping in backends module](https://github.com/django/django/blob/main/django/db/backends/postgresql/base.py). Below is the mapping taken from source code.

```python
class DatabaseWrapper(BaseDatabaseWrapper):
    vendor = 'postgresql'
    display_name = 'PostgreSQL'
    # This dictionary maps Field objects to their associated PostgreSQL column
    # types, as strings. Column-type strings can contain format strings; they'll
    # be interpolated against the values of Field.__dict__ before being output.
    # If a column type is set to None, it won't be included in the output.
    data_types = {
        'AutoField': 'serial',
        'BigAutoField': 'bigserial',
        'BinaryField': 'bytea',
        'BooleanField': 'boolean',
        'CharField': 'varchar(%(max_length)s)',
        'DateField': 'date',
        'DateTimeField': 'timestamp with time zone',
        'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
        'DurationField': 'interval',
        'FileField': 'varchar(%(max_length)s)',
        'FilePathField': 'varchar(%(max_length)s)',
        'FloatField': 'double precision',
        'IntegerField': 'integer',
        'BigIntegerField': 'bigint',
        'IPAddressField': 'inet',
        'GenericIPAddressField': 'inet',
        'JSONField': 'jsonb',
        'OneToOneField': 'integer',
        'PositiveBigIntegerField': 'bigint',
        'PositiveIntegerField': 'integer',
        'PositiveSmallIntegerField': 'smallint',
        'SlugField': 'varchar(%(max_length)s)',
        'SmallAutoField': 'smallserial',
        'SmallIntegerField': 'smallint',
        'TextField': 'text',
        'TimeField': 'time',
        'UUIDField': 'uuid',
    }
```

`get_internal_type` values are keys and values are Postgres field names.

I have skipped the implementation of the rest of the methods like `__reduce__` and `check` . You can go through the source code of Django fields in [GitHub](https://github.com/django/django/blob/main/django/db/models/fields/__init__.py) and also you will find class variables and private methods usages.

Django documentation has an excellent page on [how-to write custom model field](https://docs.djangoproject.com/en/3.2/howto/custom-model-fields/).

## Summary

1. `models.Field` is the root of all the model fields.
2. Field initializer takes configuration details like `name, default, db_index, null` for the database columns and `blank, help_text` for non-column features like Django model form and Django admin.
3. `__init__` method in the child class can override the user passed value and(or) set custom default value.
4. `validators` attribute in the field contains the user-defined validators and default validators specific to the field.
5. Every field needs to implement a few methods to work with the specific databases. Some of the useful methods are `to_python, get_db_prep_value, get_prep_value, deconstruct, formfield,  db_type`.
6. Django `Connection` object or wrapper contains details and features of the underlying the database.

## Notes
- Cover Photo by <a href="https://unsplash.com/@ninjason>Jason Leung</a> on <a href="https://unsplash.com/s/photos/ingredients">Unsplash</a>
- The post first appeared in [my blog](https://kracekumar.com/post/django_model_fields_orm_part2/).