Build a Live Search Feature with Django using HTMX and PostgreSQL FTS (in 10 minutes)
Tags: python, django, postgres
Published: 2024-04-06 11:46:41

We're going to build a simple app that uses advanced PostgreSQL full-text search features, with results that update as the user enters their query.

This includes:

* using HTMX to fetch results as the user types (without extra JavaScript)
* allowing the user to enter search engine style queries (i.e. "python or django")
* adding a Django management command to quickly insert test data

Here's an image of the working product:

![Full text search example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7pou9dolu7590c33cv3i.png)

### Previewing the project

Want to see a live version of the app?  You can create a copy of this project now and try it out.

<a href="https://dashboard.circumeo.io/projects/t9z1nbun/fork/" target="_blank">Copy the FullTextSearchApp Project</a>

### Setting up the Django app

Install packages and create the Django application.

```bash
pip install --upgrade django faker
django-admin startproject eventsearch .
python3 manage.py startapp core
```

Add `core` to the `INSTALLED_APPS` list.

```python
# settings.py
INSTALLED_APPS = [
  "core",
    ...
]
```

### Adding the templates

* Create a directory named `templates` within the `core` app.
* Create a file named `search.html` within the `templates` directory.

```html
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <script src="https://unpkg.com/htmx.org"></script>
        <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css"
              rel="stylesheet"
              integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC"
              crossorigin="anonymous">
        <link href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.0/font/bootstrap-icons.css"
              rel="stylesheet">
        <style>
      html,
      body {
        height: 100%;  
      }
      
      #content {
        display: flex;
        flex-direction: column;
        height: 100%;
      }
      
      #events-content {
        background-color: rgb(246, 247, 248);
        flex: 1;
      }

      #events-content-inner {
        max-width: 600px;
        width: 100%;
      }

      #event-search-input {
        border-top-left-radius: 8px;
        border-bottom-left-radius: 8px;
        border: 1px solid rgb(217, 217, 217);
        border-right: none;
        max-width: 600px;
        outline: none;
      }

      #search-submit-btn {
        background-color: rgb(246, 88, 88);
        border-top-right-radius: 8px;
        border-bottom-right-radius: 8px;
        border: none;
        color: #fff;
        width: 42px;
      }
          
      .search-result {
        border-bottom: 1px solid #ccc;
      }

      .search-result-inner {
         display: flex;
         gap: 1rem;
      }

      .event-thumbnail {
         border-radius: 8px;
         width: 160px;
         height: 100px;
      }

      .event-description {
         font-size: 13px;  
      }
          
       .event-date {
         color: rgb(124, 111, 80);
         font-weight: 500;
       }
        </style>
    </head>
    <body>
        <div id="content">
            <form method="post">
                {% csrf_token %}
                <div id="event-search" class="d-flex justify-content-center p-3">
                    <div class="d-flex justify-content-center w-50">
                        <input id="event-search-input"
                               name="q"
                               class="w-100 p-2"
                               type="text"
                               value="{{ query }}"
                               placeholder="Search events"
                               hx-post="/"
                               hx-select="#events-content-inner"
                               hx-swap="outerHTML"
                               hx-trigger="input changed delay:500ms, search"
                               hx-target="#events-content-inner" />
                        <button id="search-submit-btn" type="submit">
                            <i class="bi bi-search"></i>
                        </button>
                    </div>
                </div>
            </form>
            <div id="events-content" class="d-flex justify-content-center">
                <div id="events-content-inner" class="p-4">
                    {% for event in events %}
                        <div class="search-result py-4">
                            <div class="search-result-inner">
                                <div>
                                    {% if event.image_url %}<img class="event-thumbnail" src="{{ event.image_url }}" />{% endif %}
                                </div>
                                <div>
                                    <div class="event-date">{{ event.event_date|date:"l, F d" }} {{ event.event_time }}</div>
                                    <div>{{ event.name }}</div>
                                    <div class="event-description mt-2">{{ event.description }}</div>
                                </div>
                            </div>
                        </div>
                    {% empty %}
                    <div class="mx-auto text-center py-4">
                      <p>No events found.  Have you run the management command yet?</p>  
                      <p>Click Shell in the IDE and then Connect.</p>
                      <pre class="mt-5">python3 manage.py populate_events</pre>
                    </div>
                    {% endfor %}
                </div>
            </div>
        </div>
    </body>
</html>
```

### Adding the views

Copy and paste the following into `views.py` within the `core` directory.

```python
from django.contrib.postgres.search import SearchQuery
from django.shortcuts import render
from core.models import Event


def search(request):
    events, q = [], ""

    if request.method == "POST":
        q = request.POST.get("q")
        if q:
            events = Event.objects.filter(
                description_vector=SearchQuery(q, search_type="websearch")
            )

    if not events:
        events = Event.objects.all()

    return render(request, "core/search.html", {"query": q, "events": events})
```

### Updating URLs

Create `urls.py` in the `core` directory.

```python
from django.urls import path
from .views import search

urlpatterns = [
    path("", search, name="search")
]
```

Update the existing `urls.py` within the project `eventsearch` directory.

```python
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]
```

### Adding the database models

Overwrite the existing `models.py` with the following:

```python
from psycopg2.extensions import register_adapter, AsIs

from django.db import models
from django.db.models import F
from django.contrib.auth.models import User

from django.contrib.postgres.search import SearchVectorField
from django.contrib.postgres.indexes import GinIndex

class PostgresDefaultValueType:
    pass

# Wrap the built-in SearchVectorField so that we pass DEFAULT when value is None.
# Otherwise, the Django ORM will try to update the `description_vector` field, which
# PostgreSQL won't allow because it is a GENERATED column.

# Credit for this approach belongs here:
# https://code.djangoproject.com/ticket/21454#comment:28
class DelegatedSearchVectorField(SearchVectorField):
    def get_prep_value(self, value):
        return value or PostgresDefaultValueType()

register_adapter(PostgresDefaultValueType, lambda _: AsIs('DEFAULT'))

class Event(models.Model):
    name = models.CharField(max_length=100)
    description = models.TextField()
    description_vector = DelegatedSearchVectorField()
    
    image_url = models.URLField(max_length=1024, blank=True, null=True)
    event_date = models.DateField()
    event_time = models.TimeField()
```

### Adding a Manual Migration for the Search Vector Field

Run the migration command to generate a new empty migration.

```bash
python3 manage.py makemigrations core --empty
```

Populate the empty migration with the following.  

```python
from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('core', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            """
            ALTER TABLE core_event ADD COLUMN description_vector tsvector GENERATED ALWAYS AS (
                setweight(to_tsvector('english', coalesce(name, '')), 'A') ||
                setweight(to_tsvector('english', coalesce(description, '')), 'B')
            ) STORED;
            CREATE INDEX IF NOT EXISTS description_vector_gin ON core_event USING GIN(description_vector);
            """,
            reverse_sql=
            """
            ALTER TABLE core_event DROP COLUMN description_vector;
            """,
        ),
    ]
```

The migration will create an automatically updated `tsvector` by using a `GENERATED` column.  Django 5.0 supports the <a href="https://docs.djangoproject.com/en/5.0/ref/models/fields/#generatedfield" target="_blank">GeneratedField</a> type, but the SQL expression is too complex to handle that way, so we'll use a manual migration to handle it without the ORM. 

### Adding a Django Management Command for Test Data

* Create the directory structure `management/commands` within the `core` folder.
* Open a file named `populate_events.py` within the new directory and enter the following.

```python
from datetime import datetime, timedelta
import random

from django.core.management.base import BaseCommand
from django.contrib.auth.models import User
from django.utils.timezone import make_aware
from faker import Faker

from core.models import Event

class Command(BaseCommand):
    def handle(self, *args, **options):
        self.stdout.write(self.style.SUCCESS('Starting database population...'))

        faker = Faker()

        for i in range(10):
            first_name = faker.first_name()
            last_name = faker.last_name()
            User.objects.create_user(username=f'user{i}',
                                     email=f'user{i}@example.com',
                                     password='password',
                                     first_name=first_name,
                                     last_name=last_name)

        event_names = [
            'Django for Beginners',
            'Python Data Science Meetup',
            'React Native Workshop',
            'Machine Learning Basics',
            'Introduction to IoT',
            'Blockchain 101',
            'Advanced Django Techniques',
            'Full-stack Development with Python and JavaScript',
            'Mobile App Development Trends',
            'Cybersecurity Fundamentals',
        ]

        descriptions = [
            'Learn Django through practical examples, from setting up your environment to building and deploying a web application.',
            'Explore data science techniques using Python, including statistical analysis, machine learning algorithms, and data visualization.',
            'Build your first React Native app by learning the basics of this cross-platform framework and creating a simple application.',
            'An introduction to machine learning concepts, covering foundational principles, algorithms, and real-world applications.',
            'Getting started with the Internet of Things (IoT): Learn how to connect devices and collect data for analysis and insights.',
            'Understanding blockchain technology, from the basics of decentralized networks to smart contracts and cryptocurrency.',
            'Deep dive into Django features, exploring advanced functionalities that can help in building robust and scalable web applications.',
            'Learn full-stack development from front to back, covering everything from basic HTML/CSS to backend programming and databases.',
            'What\'s new in mobile app development: Explore the latest trends, tools, and technologies shaping the future of mobile applications.',
            'Protect your digital assets with cybersecurity basics, learning about threats, safeguards, and best practices for online security.',
        ]

        image_urls = [
          'https://circumeo.io/static/images/server-racks.png',
          'https://circumeo.io/static/images/typewriter.jpg',
          'https://circumeo.io/static/images/library.jpg',
          'https://circumeo.io/static/images/landscape.jpg',
          'https://circumeo.io/static/images/laptop.jpg',
          'https://circumeo.io/static/images/latte.jpg',
        ]

        for name, description in zip(event_names, descriptions):
            image_url = random.choice(image_urls)
            
            event_date = datetime.now() + timedelta(days=random.randint(1, 90))
            event_time = datetime.now().time()
            
            Event.objects.create(name=name, description=description, image_url=image_url, event_date=make_aware(event_date), event_time=event_time)

        self.stdout.write(self.style.SUCCESS('Database population completed successfully!'))
```

Open a shell session in order to run the Django management command.

```bash
python3 manage.py populate_events
```

### Up and Running with Full Text Search

That's it!  You now have an app up and running that incorporates powerful full-text search capabilities.  Not to mention, the search is dynamic as well, without the need for additional JavaScript, thanks to HTMX.