Developing Maintainable Django Projects for the Long Run
Tags: django, webdev, python
Published: 2021-08-01 22:21:43

Django being the web framework for the perfectionists definitely comes with many handy features out of the box. Even after being a very opinionated framework [there is always the "Django way" for doing things], in many cases there are more than a few ways to do the same thing. This might lead to some poor structure and coding choices for many beginners as described in [this blog post](https://www.toptal.com/django/django-top-10-mistakes). These mistakes [along with many others] often lead to unmanageable code in the long run when the size of the project keeps growing.

If there is a bug and causes downtime in production then fixing the error might feel like finding a needle in the haystack with a poorly structured codebase. The above mentioned link can work as a style guide [like many other, scroll to the end of this blog post for more Django style guides] for structuring Django projects which can help manage our code better. On top of this, let's dive into 11 ways that will definitely make life easier developing and maintaining a Django project for the long run.


## 1. Code Formatting and Linting
The codebase should be clean, concise and readable as suggested by the [Django Best Practices](https://django-best-practices.readthedocs.io/en/latest/code.html) guide. To maintain such a codebase, the best approach is to follow a specific code style and format. The hard part is remembering all the formatting rules and getting every developer working on the project onboard with these specific rules. That can eat up a lot of unnecessary time in development.

Enter [black](https://github.com/psf/black) - The Uncompromising Code Formatter. It supports all major IDEs and upon saving a file, it does it's black magic to convert all the messy code to beautifully structured code. You can also run black against your whole codebase for formatting all of the files as well. Here is a [handy guide on dev.to](https://dev.to/adamlombard/how-to-use-the-black-python-code-formatter-in-vscode-3lo0) for setting up black in VSCode.

To help you with syntax errors, [pylint](https://github.com/PyCQA/pylint) is an amazing tool. There is also a dedicated Django extension for pylint called [pylint-django](https://github.com/PyCQA/pylint-django). Pylint is the default linter in VSCode and you can setup pylint-django just by installing it via pip and updating the settings as shown on [here](https://code.visualstudio.com/docs/python/linting#:~:text=Pylint%20plugins%2C%20such,load-plugins%22%2C%20%22pylint_django%22%5D).

Finally, to help you code even faster, you can use [Kite](https://www.kite.com/) for AI powered auto-completion.

## 2. Environment Variables and Django Settings File
Your environment variables like `SECRET_KEY` or anything related to database setup should never be placed directly in the Django settings file and in the version control. These are sensitive data and you should safely keep them out of anyone's reach who's not supposed to see it.

You can use [django-environ](https://github.com/joke2k/django-environ) for serving the purpose. The process is very straightforward as well. You just need to create an `.env` file and keep all your variables there and import them into your settings file using the API of django-environ package. And keep this `.env` file out of version by placing it in the `.gitignore` file. But you should keep an `.env.example` file in the version control that contains a template of the original `.env` file.

Your development coding environment will obviously differ in one way or another and to match with that, your project's settings file will differ as well. If you are keeping your development settings in the old fashioned `local_settings.py` file and keeping out of version control, you are probably doing it wrong and there are many [valid reasons for that](https://daniel.feldroy.com/posts/using-executable-code-outside-version-control).

A better approach is to split up your settings file into base, development, test, stage and production files along with a `__init__.py` file that will go to a settings directory replacing the `settings.py` file. The `base.py` settings file will hold all the settings that don't need to change and the development, test, stage and production settings files containing all the environment specific settings with start with importing all the settings from the `base.py` file as `from .base import *`.

One issue that will come up with this approach is that the `manage.py` will be pointing to the older settings file and it'll no longer work. In this case you won't be able to run your development server along with any commands with `manage.py`. To solve this, you can create a new environment variable called `WORK_ENV` in your `.env` file and a new `engage.py` file in the settings directory which will contain -

```
from .base import *

working_environment = env.str("WORK_ENV", default="development")


if working_environment == "production":
    from .production import *
elif working_environment == "stage":
    from .stage import *
elif working_environment == "test":
    from .test import *
else:
    from .development import *
```
And finally, update your manage.py file to replace the line 

```os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project.settings')```
with 
```os.environ.setdefault("DJANGO_SETTINGS_MODULE", "your_project.settings.engage")```

So, previously all the settings with all sensitive information would have been stored in a single `settings.py` file. And now, with the split up settings file and the `.env` file, it should look like -

```
|--settings
   |-- __init__.py
   |-- base.py
   |-- engage.py
   |-- development.py
   |-- production.py
   |-- stage.py
   |-- test.py
   |-- .env
   |-- .env.example
```

## 3. PostgreSQL Everywhere 
Django comes with a simple and lightweight SQLite3 database out of the box which takes you out of the hassle of setting up the database and you can go straight to developing your application. But there are many cases where it might go wrong and even restrict you from writing efficient code for filtering your data from the database. Among others, one example can be how to filter distinct values as explained [here](https://stackoverflow.com/questions/4723220/retrieving-distinct-records-based-on-a-column-on-django#comment23664405_4725945), you can't really use the SQLite3 database for this.

PostgreSQL database works fantastic with Django. It is [recommended](https://djangodeployment.com/2016/12/23/which-database-should-i-use-on-production/) by many to use PostgreSQL in production. A better approach would be using it in all of your environments [development, test, stage and production] so that your application works consistently everywhere. If you can just go through the setting up the PostgreSQL database on your environments [just a few commands anyway and if you use Docker you can streamline the process even further], you don't have to spend time thinking about which Django ORM features that you can use.

## 4. API Documentation
If you are working with Django REST framework, it's a good idea to keep your API documented even if you are not exposing it to the public. One great tool is [Swagger](https://swagger.io/) and you can generate the Swagger schema automatically for your DRF project by using [drf-yasg](https://github.com/axnsan12/drf-yasg). Very easy to setup and the schema will be generated from your `urls.py` files. You can also choose between two different looks.

Keep in mind that drf-yasg generates Swagger/OpenAPI 2.0 specification of your REST API. If you are looking for Swagger/OpenAPI 3.0, you can check out [drf-spectacular](https://github.com/tfranzel/drf-spectacular).

## 5. Django Debug Toolbar 
The [django-debug-toolbar](https://github.com/jazzband/django-debug-toolbar) package makes debugging a breeze. Django's default debug mode is definitely a very useful tool but django-debug-toolbar is a step above. You can see how much CPU time it takes to process the response, how many [and what] SQL queries are being executed, which static files the request is fetching, details about your template, list of all of your context variables and many more.

## 6. Django Shell and Management Commands
Django shell is like a Python shell but with everything that Django is setup with on your project settings. You can call all of your apps, models and anything that's defined in your project really. Just type in `python manage.py shell` in your virtual environment and you're inside the Django shell. This can come in handy for testing different ORM queries with your defined app models as well as trying out APIs of third party packages.

You can use custom management commands with `manage.py` for doing tasks. For example, you might need to insert data into your database from a csv file or a daily cron job for doing something that is related to the database. You can't access the database with the Django ORM from outside the application [for example, a cron job]. But if you define your task in a management command, you can call it from anywhere with the Python from that project virtual environment using `python manage.py your_command`.

There are many useful management commands can already be found in different third party packages. One such package is [django-extensions](https://github.com/django-extensions/django-extensions). It comes with many helpful commands that can help you in many different scenario in development. You can drop the whole database, get a list of all of your urls and many more. One feature I particularly like is that I can generate a visual representation of my models using [this](https://github.com/django-extensions/django-extensions#:~:text=%24%20python%20manage.py%20graph_models%20-a%20-o%20myapp_models.png) command. When the codebase grows big and I need to get an overview of how all of my models are connected, it comes in very handy.
 
## 7. Logging with Sentry 
Well, [the night is dark and full of (t)errors](https://media.giphy.com/media/uh2RP33ezYpnW/giphy.gif). And you can't just play [Jon Snow](https://media.giphy.com/media/13f5iwTRuiEjjW/giphy.gif) when the users start complaining about getting server errors.

In development, when your debug mode is on, you can just go through the very helpful debug messages that Django helps you with. But it's not the same in production. You should definitely [setup logs](https://mattsegal.dev/django-gunicorn-nginx-logging.html) in your Django projects but additionally, a third party tool like [Sentry](https://sentry.io/welcome/) can save you a ton of time and hassle. The setup process can't be simpler, you just sign up and [add a few lines](https://docs.sentry.io/platforms/python/guides/django/) to the Django settings file and that's it!

To get you started, the free tier gives you to save 5K errors, 10K transactions and 1GB attachments for 1 user with 30 day data retention. And you don't need to wait for your users to start complaining that your site isn't working, Sentry will send you emails when it catches new errors.

## 8. Environment Management - pip-tools and Docker
You should always pin the exact version of Django and other pip packages that you are using in your Django project in a requirements.txt file. You can easily do it with `virtualenv` and a `pip-freeze` command. But the issue that comes up with this is that it exports all the package names that are in your virtual environment. There can be many dependencies of a specific package and if you want to upgrade one, you'll need to update all of it's dependencies manually as well along with it. Also, there are packages that you need only in development like `black`, `pylint` or `django-debug-toolbar` that you definitely don't need in production. To deal with all these, there is an awesome package called [pip-tools](https://github.com/jazzband/pip-tools).

With the help of this package, you only need to pin your packages in a `requirements.in` file and when you compile this file, it'll generate a nicely formatted `requirements.txt` file that will have all the dependencies with notes for which one of them is coming from which specific package. You can also pin your development packages in `dev-requirements.in` file and keep your development tools separate.

Now you have taken care of the version of the pip packages, how about your Python version? Or exact PostgreSQL version that you are using in your development? Or even further, the version of your operating system that's all of these are running on?

When you run your application, it's behavior depends on everything that it's using [and not using, that are running alongside it on the same operating system]. All of these can influence how it's running and performing. To isolate the application and to make sure it works the same no matter where it's running, you should use [Docker](https://www.docker.com/).

Getting started with Docker can easily feel overwhelming for a newcomer. [Here](https://testdriven.io/blog/dockerizing-django-with-postgres-gunicorn-and-nginx/) is an awesome article how you can user Docker with your Django project. This is a three part series and it walks you through how you can use Docker in your development as well as deploying it live.

## 9. Using UserAdmin in the Django Admin for Custom User Model
One of the most handy "batteries" included with Django has to be the Django Admin. It can be very useful for inspecting the data in the database with a model-centric interface as well as for inputting / inspecting / managing test data while in development. When we add a model to the Django Admin, the usual approach is to use `ModelAdmin` from `django.contrib.admin` but if you use it with a custom user model, there is an issue. And if you are not already aware, defining the custom user model should be something that you should do as one of the first things when you start a new Django project as [suggested](https://docs.djangoproject.com/en/3.2/topics/auth/customizing/#:~:text=Using%20a%20custom,sufficient%20for%20you.) by the Django documentation itself.

Just like most other web frameworks, Django doesn't store the passwords as plain texts. It uses a hash function to encrypt it and then stores it in the database. So even if the data is compromised, it won't be possible to decipher what actually the password is for a specific user.

Going back to the issue about using `ModelAdmin` with a custom user model is that your user create form and user edit form will present you with password input fields where you'll need to enter the password in the hashed format itself. If you enter _yourawesomepassword123_ as your password and save the user, that user won't be able to log in with _yourawesomepassword123_ because that's not the actual stored password. The actual password would be the value that's is _yourawesomepassword123_ after hashing.

The solution is to write user creating and updating forms based on `UserCreationForm` and `UserChangeForm` from `django.contrib.auth.forms` and use them in the admin that will be based on `UserAdmin` from `django.contrib.auth.admin`. You can find all about how to do it on [this awesome tutorial](https://learndjango.com/tutorials/django-custom-user-model). This way you will have the freedom of creating new users from the Django admin with ease.

## 10. TDD - Test Driven Development
Test driven development is where you write your tests even before you code a feature. The idea is that you will think of the test case, write a test and write the function that will make the test pass. It is certainly not easy to always follow this and getting started with it can feel overwhelming. But for the long run, it'll help you save so much hours of pain and misery for fixing unwanted bugs that happen in production.

I would suggest checking out [testdriven.io](https://testdriven.io/) for getting up and running with TDD [and Docker]. I can vouch that their paid courses are worth every penny without having any affiliation with them.

## 11. Starter Templates and Style Guides
There are a number of things that always need to be done when starting a new project which are always the same. There are a some fantastic starter templates that can help jumpstart your Django projects. I'm listing a few of them below - 
* [cookiecutter-django](https://github.com/pydanny/cookiecutter-django)
* [djangox](https://github.com/wsvincent/djangox/)
* [drfx](https://github.com/wsvincent/drfx/)
* [django-project-template](https://github.com/jpadilla/django-project-template)
* [docker-django](https://github.com/erroneousboat/docker-django/)

When the project keeps getting larger and multiple developers working on the same project [even while working alone], it is possible to end up with different style codes in the same codebase. It's better to follow a specific style guide throughout all of the codebase so it's easier to read, modify and maintain in the long run. It's also easier to get new developers onboard with the project with a specific style guide. A few of the popular styles guides - 
* [Django Best Practices](https://django-best-practices.readthedocs.io/en/latest/)
* [Django Styleguide by HackSoftware](https://github.com/HackSoftware/Django-Styleguide) which also comes with [an example project](https://github.com/HackSoftware/Styleguide-Example)
* [django-api-domains](https://github.com/phalt/django-api-domains)
* [Two Scoops of Django 3.x: Best Practices for the Django Framework](https://www.feldroy.com/products/two-scoops-of-django-3-x)