Handling Big Data with Django and AgGrid
Tags: python, django, aggrid
Published: 2024-04-10 19:53:44

We're going to build an app that can handle filtering and sorting large amounts of data with AgGrid.

Sending all of your data to the browser is a quick way to get started with AgGrid, but what if you have a million rows? That's when server-side filtering and sorting come to the rescue.

This guide includes:

* using AgGrid in infinite scroll mode so that the server is responsible for paginating, filtering, and sorting
* implementing filtering, sorting, and pagination for AgGrid on top of a Django ListView class
* creating a Django management command to populate test data

But first, here's a quick peek at what we're building today:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u8wzkgh76h0feqylat6p.png)

### Previewing the project

Want to see a live version of the app? You can create a copy of this project now and try it out.

[Create a fork of the AgGridDemo project on Circumeo.](https://dashboard.circumeo.io/projects/j587uv9l/fork/)

### Setting up the Django app

Install packages and create the Django application.

```
pip install --upgrade django faker
django-admin startproject bigdata .
python3 manage.py startapp core
```

Add core to the `INSTALLED_APPS` list.

```python
# settings.py
INSTALLED_APPS = [
  "core",
    ...
]
```

### Adding the templates

* Create a directory named `templates` within the `core` app.
* Create a file named `index.html` within the `templates` directory.

```html
<!DOCTYPE html>
<html>
  <head>
    <script src="https://unpkg.com/ag-grid/dist/ag-grid.min.noStyle.js"></script>
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-grid.css">
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-theme-balham.css">
    <style>
      #container {
        max-width: 1250px;
        width: 100%;
        height: 100%;
        margin: 0px auto;
        margin-top: 30px;
      }
    </style>
  </head>
  <body>
    <div id="container">
      <div id="data-grid" style="height: 600px; width: 100%" class="ag-theme-balham"></div>
    </div>
    <script type="text/javascript" charset="utf-8">
      const numberFilterParams = {
        filterParams: {
          filterOptions: ["equals", "notEqual", "lessThan", "greaterThan"],
          suppressAndOrCondition: true,
        },
      };

      var columnDefs = [
        { headerName: "Name", field: "name", filter: "agTextColumnFilter" },
        {
          headerName: "Description",
          field: "description",
          filter: "agTextColumnFilter",
        },
        { headerName: "Category", field: "category", filter: "agTextColumnFilter" },
        {
          headerName: "Price",
          field: "price",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Stock Quantity",
          field: "stock_quantity",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Rating",
          field: "rating",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
      ];

      var gridOptions = {
        columnDefs: columnDefs,
        defaultColDef: {
          filterParams: {
            suppressAndOrCondition: true,
          },
        },
        enableServerSideSorting: true,
        enableServerSideFilter: true,
        rowModelType: "infinite",
        cacheBlockSize: 100,
        maxBlocksInCache: 10,
      };

      var dataSource = {
        getRows: function (params) {
          var filtering = encodeURIComponent(JSON.stringify(params.filterModel));
          var sorting = encodeURIComponent(JSON.stringify(params.sortModel));

          var startRow = params.startRow;
          var endRow = params.endRow;

          var url = `/products?startRow=${startRow}&endRow=${endRow}&filter=${filtering}&sort=${sorting}`;

          fetch(url)
            .then((response) => response.json())
            .then((data) => {
            params.successCallback(data.rows, data.totalRows);
          })
            .catch((err) => {
            params.failCallback();
          });
        },
      };

      var gridDiv = document.querySelector("#data-grid");

      new agGrid.Grid(gridDiv, gridOptions);
      gridOptions.api.setDatasource(dataSource);
      gridOptions.api.sizeColumnsToFit();
    </script>
  </body>
</html>
```

### Adding the views

* Remove the existing `views.py` file.
* Create a new directory named `views` within the `core` folder.
* Create a new file named `index.py` within the `views` directory.
* Copy and paste the following into `index.py` within the `views` directory.

```python
from django.shortcuts import render

def index_view(request):
    return render(request, "core/index.html")
```

* Create another file named `products.py` within the same directory.
* Copy and paste the following into the `products.py` file.

```python
import json

from django.views.generic.list import ListView
from django.http import JsonResponse
from django.db.models import Q

from core.models import Product


class ProductListView(ListView):
    model = Product

    def get_queryset(self):
        """
        Convert AgGrid filter and sort objects into a Django query.
        An example filter:
            {"category": {"type": "contains", "filter": "electronics"}}
        """
        queryset = super().get_queryset()

        filter_params = self.request.GET.get("filter", None)
        if filter_params:
            filters = json.loads(filter_params)
            q_objects = Q()

            for key, filter_info in filters.items():
                filter_type = filter_info.get("type")
                filter_value = filter_info.get("filter")

                if filter_type == "contains":
                    lookup = f"{key}__icontains"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "equals":
                    lookup = f"{key}__exact"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "notEqual":
                    lookup = f"{key}__exact"
                    q_objects &= ~Q(**{lookup: filter_value})
                elif filter_type == "greaterThan":
                    lookup = f"{key}__gt"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "lessThan":
                    lookup = f"{key}__lt"
                    q_objects &= Q(**{lookup: filter_value})

            queryset = queryset.filter(q_objects)

        sort_params = self.request.GET.get("sort", None)
        if sort_params:
            sort_objects = json.loads(sort_params)
            sort_fields = []
            for sort_object in sort_objects:
                col_id = sort_object["colId"]
                sort_order = sort_object["sort"]
                if sort_order == "asc":
                    sort_fields.append(col_id)
                elif sort_order == "desc":
                    sort_fields.append(f"-{col_id}")

            if sort_fields:
                queryset = queryset.order_by(*sort_fields)
        return queryset

    def get(self, request, *args, **kwargs):
        start_row = int(request.GET.get("startRow", 0))
        end_row = int(request.GET.get("endRow", 100))

        queryset = self.get_queryset()

        total_rows = queryset.count()
        queryset = queryset[start_row:end_row]

        products = list(
            queryset.values(
                "name", "description", "category", "price", "stock_quantity", "rating"
            )
        )

        return JsonResponse({"rows": products, "totalRows": total_rows})
```

### Updating URLs

* Create urls.py in the core directory.

```python
from django.urls import path

from core.views.index import index_view
from core.views.products import ProductListView

urlpatterns = [
    path("", index_view, name="index"),
    path("products", ProductListView.as_view(), name="products"),
]
```

* Update the existing urls.py within the project bigdata directory.

```python
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]
```

### Adding the database models

Overwrite the existing `models.py` with the following:

```python
from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=255)
    description = models.TextField(blank=True)
    category = models.CharField(max_length=255)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    stock_quantity = models.IntegerField()
    rating = models.FloatField(blank=True, null=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
```

### Adding a Django Management Command for Test Data

* Create the directory structure `management/commands` within the `core` folder.
* Open a file named `populate_products.py` within the new directory and enter the following.

```python
from faker import Faker
import random

from django.core.management.base import BaseCommand
from core.models import Product

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        faker = Faker()

        categories = [
            'Electronics',
            'Books',
            'Clothing',
            'Home & Garden',
            'Toys & Games',
            'Sports & Outdoors',
            'Health & Beauty',
            'Automotive',
            'Groceries',
            'Pet Supplies'
        ]

        for _ in range(1000):
            Product.objects.create(
                name=faker.text(max_nb_chars=20).capitalize(),
                description=faker.text(),
                category=random.choice(categories),
                price=round(random.uniform(5.0, 500.0), 2),
                stock_quantity=random.randint(0, 100),
                rating=round(random.uniform(1.0, 5.0), 1)
            )

        self.stdout.write(self.style.SUCCESS('Successfully populated the database with products.'))
```

Open a shell session in order to run the Django management command.

```
python3 manage.py populate_products
```

### Up and Running with Django and AgGrid

You're now ready to handle millions of rows without crashing browser tabs! Add database indexes where needed, and you can scale this approach pretty far.