Data Analysis Setup on Django data on Single machine
Tags: django, postgres, devops, cgroup
Published: 2020-04-10 15:29:01


We at my workplace use Django based backend and frequently need to perform data analysis tasks on the data in the django models. This post provides a basic setup for allowing data analysts to analyse data using the jupyter notebook.

Note: This can be done in a lot better way using docker, jupyterlab, separate machines etc, there is no limit. This post just aims to outline the setup which was done in the constraints and timeline available.

## Tools Used

[shell_plus](https://django-extensions.readthedocs.io/en/latest/shell_plus.html) - Django shell with all models imported already in a jupyter notebook and more.

[supervisor](http://supervisord.org/) - to run the notebook server continuously in background

[postgresql](https://www.postgresql.org/) - Database. Creating a read only user so that analysts can work on latest production db without risk of changing it.

[cgroups](https://en.wikipedia.org/wiki/Cgroups) - For making sure that notebook only consumes limited amount of CPU and memory

## Use django_extensions

Install django_extensions as an app in your django project. Instructions for installation and usage are available on [https://django-extensions.readthedocs.io/en/latest/](https://django-extensions.readthedocs.io/en/latest/).

Once done you will be able to run and test the notebook via

    ./manage.py shell_plus --notebook

## Setup a readonly user for postgres

Help take from [https://stackoverflow.com/a/42044878/1660759](https://stackoverflow.com/a/42044878/1660759) and [https://stackoverflow.com/a/54193832/1660759](https://stackoverflow.com/a/54193832/1660759)

    CREATE ROLE Read_Only_User WITH LOGIN PASSWORD 'Test1234' NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION VALID UNTIL 'infinity';

**Statements after this need to be run while connected to YourDatabaseName**

    GRANT CONNECT ON DATABASE YourDatabaseName TO Read_Only_User;
    GRANT USAGE ON SCHEMA public TO Read_Only_User;
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO Read_Only_User;
    GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO Read_Only_User;
    #So that new tables are in this database are automatically accessible to this user
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO Read_Only_User; 
    #for functions
    GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public TO Read_Only_User;

## Setup separate settings file

[Or manage through environment variables. If that is what you do you know how to do that for this example👌]

Create a new settings file, import settings from main settings file and replace the database settings with new readonly user

settings_database_readonly.py

    from .settings import *
    
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': '<db_name>',
            'USER': '<readonly_username>',
            'PASSWORD' : '<readonly_user_password>',
            'HOST': '<db_address>',
            'PORT': '<db_port>',
        }
    }
    
    NOTEBOOK_ARGUMENTS = [
        '--port', '9999',
    ]

## Limit Usage of notebook's process CPU and RAM

For limiting usage of ram and cpu I explored cpu_limit and nice but cpu_limit does not give memory and nice does not give granular control over limits. After some research I finalised on cgroups.

cgroups has very less posts around there, and i didn't want to write just some commands and rather use a config file. This decision turned out not simple enough to follow through given the documentation out there.

Comparison between cpulimiti, nice and cgroups - [https://scoutapm.com/blog/restricting-process-cpu-usage-using-nice-cpulimit-and-cgroups](https://scoutapm.com/blog/restricting-process-cpu-usage-using-nice-cpulimit-and-cgroups)

Guide for using cgroups using config file [https://www.paranoids.at/cgroup-ubuntu-18-04-howto/](https://www.paranoids.at/cgroup-ubuntu-18-04-howto/)

    apt install cgroup-tools

Copy conf file from examples

    cp /usr/share/doc/cgroup-tools/examples/cgred.conf /etc/

/etc/cgconfig.conf - limiting 10% CPU Usage and 1G memory

    group notebooks {
         cpu {
             cpu.cfs_quota_us=10000;
         }
         memory {
             memory.limit_in_bytes = 1024m;
         }
    }

/etc/cgrules.conf - This will limit every process of the processes running in group notebooks to 10% CPU and 1G of memory.

    #<user>    <controllers>           <destination>
    notebooks       cpu,memory              notebooks

For testing use this commands:

    /usr/sbin/cgconfigparser -l /etc/cgconfig.conf
    /usr/sbin/cgrulesengd -vvv

## Supervisor

Setup supervisor - [supervisor](http://supervisord.org/).

Use the following configuration in supervisor.conf for running the notebook using cgroup

    [program:readonly_shell_notebook]
    environment=DJANGO_SETTINGS_MODULE="dashboard.settings_database_readonly"
    command=cgexec -g cpu,memory:notebooks /home/videoken-engage/.virtualenvs/django_project/bin/python ../manage.py shell_plus --notebook
    directory=<directory of django project>
    stdout_logfile=<path_to_log_file>
    stderr_logfile=<path_to_log_file>

cgexec command is used to run any process in a given cgroup.

Due to using configuration files for cgroups, a file is sometimes not given proper permissions. I was getting the following error while running the cgexec command - 

    Cgroups error: cgroup change of group failed

 This fixed the problem

    chown <my_ubunut_username> /sys/fs/cgroup/memory/notebooks/tasks
    chown <my_ubuntu_username> /sys/fs/cgroup/cpu/notebooks/tasks

PS: This was my first usage of cgroups after the initial reading I did for understanding of Docker internals. I might be overlooking some concepts, any inputs regarding that is specifically welcome 👐.