Lessons Learned Building a Full-stack Framework for Django
Tags: django, javascript, python, unicorn
Published: 2020-12-13 04:04:19

The idea for [django-unicorn](https://www.django-unicorn.com) all started innocently enough with a tweet on July 8, 2020.

After jealously watching demos of [Phoenix's LiveView](https://github.com/phoenixframework/phoenix_live_view), I built a prototype of a [real-time monitoring dashboard for Django](https://github.com/adamghill/django-live-dashboard) with websockets and [Alpine.js](https://github.com/alpinejs/alpine). After a past side-project got a little derailed (read: became not fun) using Django, Vue.js, and GraphQL, the simplicity of Alpine's model struck a nice middle-ground.

Then, I noticed the author's [Livewire](https://github.com/livewire/livewire) project. Even though it was for the PHP web framework [Laravel](https://laravel.com) it sounded intriguing and I was immediately smitten with the documentation site. The thought of simplifying web development by enabling server-side code to be "called" from the front-end was appealing. Instead of building a suite of APIs, mapping data models to their REST representation, and switching languages to build a single-page app in Javascript, `Livewire` leverages backend code and provides the glue for the frontend to interact with. This fulfilled a need I saw all over the place -- it certainly isn't ideal for every application, but probably useful for 80% of the websites out there.

After watching the available screencasts I really wanted to at least prototype a project with it. But... _not_ enough to switch away from my typical tech stack of Python and Django. `Laravel` looks nice, but I am pretty invested in the Python ecosystem. So, I tweeted complaining that Django didn't have a similar library and my friend, [Michele](https://twitter.com/mjwhansen), then replied with the magic question: "why don't you make it yourself".

![Twitter post complaining that Livewire is not available for Django](https://dev-to-uploads.s3.amazonaws.com/i/y3fx0pvadmk9g20mrlfa.png)

I spent the next 3 days re-watching the `Livewire` screencasts pretty intently to see the "shape" of the JSON request and response messages, scouring the documentation site, and reading through the Livewire Javascript code to understand how it worked. My first push to Github was July 11th -- three days after that first tweet.

I remember how **magical** it felt to type into a textbox and then have Django render it in almost realtime as a regular Django template variable. Since Django's unofficial mascot is a pony, `django-unicorn` seemed like a fitting enough name for this little library I was starting to mildly obsess about.

There have been a lot of learnings over the last five months. I'll cover a few related to Python, Javascript, and then some general thoughts now that `django-unicorn` has grown up a little bit (version 0.11.0 was just released).

# Python

Python has been my preferred programming language for the past 8 years or so, and Django has been my steady go-to web framework. There might be some flashier web frameworks around, but for the raw speed I can go from idea to database tables to server-rendered HTML I wouldn't pick anything else.

## `importlib`
[importlib.import_module](https://docs.python.org/3/library/importlib.html#importlib.import_module) is the mechanism to dynamically import Python modules. `django-unicorn` uses this functionality to be able to find and load the component based on the component name's string representation that is specified in the template.

```html
{% unicorn 'hello-world' %}
```

The `hello-world` component name is converted to a module name of `hello_world.py` and class name of `HelloWorldView`. Then different Django apps are searched to find the correct module (defaults to `unicorn.components.hello_world.py`). Once the whole string is created, `import_module` is called to retrieve the correct component.

## `inspect`

Python contains a wealth of information about the code that is running... if you know where to look. The [inspect](https://docs.python.org/3/library/inspect.html) module provides a wealth of information about classes and its methods which I use to inspect for publicly available methods and fields to include in the Django template context.

## `literal_eval`

`django-unicorn` supports calling methods from the frontend with Python objects as arguments.

```html
<div u:model="dictionary">
    dictionary.name: {{ dictionary.name }}<br />
    <button u:click='set_dictionary({"name": 1, "nested": {"name": 2}})'>set dictionary</button>
</div>
```

The method arguments look like Python, but are actually strings because all interactions are via JSON. The argument in `set_dictionary({"name": 1, "nested": {"name": 2}})` needs to be parsed. Originally, I built a simple parser to convert strings into Python objects, but then stumbled onto [literal_eval](https://docs.python.org/3/library/ast.html#ast.literal_eval) which "can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself." It "may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None", but I end up manually handling `datetime` and `UUID` as well. Much safer than calling `eval()` and more sane than trying to handle all the cases yourself.

## `lru_cache`

The standard library provides the [lru_cache](https://docs.python.org/3/library/functools.html#functools.lru_cache) decorator that saves the results from up to the _maxsize_ function calls. Once the _maxsize_+1 unique function argument is called, the cache evicts the first object that got pushed into it. [cachetools](https://pypi.org/project/cachetools/) provides similar functionality as a class so it can be used without the function decorator.

LRU caches are used in `django-unicorn` to prevent re-finding and re-constructing component classes, and to prevent re-serializing the same data from a Python dictionary to a string representation. Both processes can be comparatively slow and tend to happen multiple times with the same inputs.

## `typing`

Typing is relatively new to the Python ecosystem (introduced with [PEP 484](https://www.python.org/dev/peps/pep-0484/)), but I find them to be a useful addition, especially with [mypy](https://pypi.org/project/mypy/) and an editor that understands the type annotations (personally, I have been pretty happy with `VS Code` and [pylance](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance)).

Optional type annotations are only designed to help developers understand the code (they aren't used by the runtime for optimization -- at least not yet), but even so they have been useful for "future me" to better understand the context for my code. Coming from a previous static language, C#, I appreciate types to a certain degree, but I find this middle-ground to be particularly useful -- I have the freedom to prototype without a rigid type system in place, but as the design solidifies I tend to add in appropriate types where they might be useful.

## Other helpful Python third-party packages

- [orjson](https://pypi.org/project/orjson/): chosen because 1) it appears to be one of the fastest JSON serializers for Python, and 2) it provides library support for serializing more datatypes than the out-of-the-box `json` library (plus, it provides a hook to "dump" other types as needed)
- [shortuuid](https://pypi.org/project/shortuuid/): used to create a unique identifier for components. The potential for possible collision is acceptable because of the limited potential number of components
- [wrapt](https://pypi.org/project/wrapt/): decorators are easy to create, but deceptively hard to make correct and `wrapt` handles all the hard parts
- [beautifulsoup4](https://pypi.org/project/beautifulsoup4/): sanely parse HTML without tearing your hair out

# Javascript

I have primarily been a backend developer for most of my career and, besides a few side projects written in the early years of Node.js, I haven't worked substantially in Javascript besides adding little features here or there. Yet, as Michael Abrahamsen writes in his [post about Flask-Meld](https://michaelabrahamsen.com/posts/flask-meld-ditch-javascript-frameworks-for-pure-python-joy/), "...here I am, writing a whole lot of JavaScript so that I can write less JavaScript. I am an engineer, after all." It's a funny thought and I wouldn't say that I'm a particularly great Javascript developer, but I have learned a ton over the past 5 months about the DOM and the more "modern" Javascript ecosystem.

## `ES6`

I'm not ashamed to say it: for a long time I didn't "get" the reasons to use anything other than ES5 Javascript on the browser for a long time. I didn't understand why I needed classes when I could do the limited DOM interactions I needed with prototype inheritance and functions. I also chafed at what I assumed was the requirement to use a transpiler like `babel` for what seemed like such minimal benefits.

In fact, when I first started `django-unicorn` it was all written in ES5 with lots and lots of unwieldy functions everywhere. Over time, it became really hard to follow the flow of code between all of the functions and I coudn't organize the code into understandable parts. Maybe it's just the "object oriented" mindset that has been drilled into me over the years, but I found Javascript `modules` and the ability to use `class` to be  extremely useful to organize the code.

## `babel` and `rollup`

Since the modern browser support is almost universal for the ES6 features I use (95%+) I can develop using ES6 Javascript  and only transpile to ES5 when generating the minified version that I ship in the library. Originally, I just fought with [babel](https://babeljs.io), but quickly afterwards I looked for something easy to configure that could also minify the separate Javascript files into one file.

After looking at `Livewire` code again, I realized they use [rollup](https://rollupjs.org/) which looked like it would fit the bill. There was quite a bit of fiddling and reading about [IIFE](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) to understand what I wanted to do, but now the build process is quick and painless.

## `ESBuild` (potentially)

I also investigated [esbuild](https://esbuild.github.io) because of the promise of even quicker build times. There is a [ESBuild PR](https://github.com/adamghill/django-unicorn/pull/65) that seems to work as expected. It even creates a slightly smaller file size than `rollup`. However, there doesn't seem to be a way to integrate `babel` into the process and I am not quite ready to give up on ES5 for users on really old browsers. At some point, that trade-off will probably shift I expect, though.

## `ava` and `jsdom`

I wanted a Javascript unit test framework that was fast and low ceremony (I think I see a recurring pattern) and [ava](https://github.com/avajs/ava) seemed the best option. `ava` has been working great so far and fits my approach well.

One issue with testing Javascript is abstracting away the DOM so you don't end up needing functional tests like `selenium` (although I spent some time with [web-test-runner](https://modern-web.dev/docs/test-runner/overview/) and [playwright](https://www.npmjs.com/package/playwright) and they were impressively fast to spin up). However, [jsdom](https://www.npmjs.com/package/jsdom) allows my tests to have enough of a DOM to test interactions like click events without requiring an actual web browser running. I did have to add hooks so that certain parts of my Javascript could use the `jsdom` DOM instead of the browser's DOM, but after that was added it seems to work well.

## `morphdom`

Part of the magic of `django-unicorn` is how the DOM gets updated. That only works reasonably well because of the work of [morphdom](https://github.com/patrick-steele-idem/morphdom). A super impressive library and also a core part of `Livewire`, as well.

# In general

Creating an open-source library isn't all about the code (as much as I really wish it was). I learned a few things that weren't related to either Python or Javascript, but about the entire process.

## Start small

`django-unicorn` started as a germ of an idea, "Livewire for Django", but _that_ was a daunting task. However, I knew what I considered the core functionality the library should have and could "see" a way to make it happen from the beginning:

- custom Django template tag that finds a component class and instantiates it
- expose all public fields on a component class to the Django template context
- Django view method that accepts a JSON object with a defined API
- Javascript that listens to events, converts them into a JSON object, and calls the Django view endpoint
- Django pieces to wrap everything together into an app that could be installed

The overall goal was overwhelming, but my [first commit](https://github.com/adamghill/django-unicorn/commit/27e2e69fc02dea587bf0b90949dc2460c96233dd) was relatively simple. I started with the basic functionality and iterated to add more and more functionality over time.

Breaking up a large project into smaller, achievable pieces is the best (or maybe only?) way I know of to build daunting, complicated software.

## Everything needs marketing

As much as open-source is lauded as some idealized meritocracy, it really isn't. Building something cool and just waiting for people to find it is an exercise in frustration. I think [Caleb Porzio](https://calebporzio.com) does a great job of this with `Livewire`. Between "working in public" on Twitter, conference talks, interviews, and podcasts, it's obvious he understands how important marketing is for his numerous projects.

The whole `Livewire` website is also marketing. The first page is basically a landing page "selling" the library and why you should use it. The API documentation is clear and concise and the _coup de grâce_ is the [screencasts](https://laravel-livewire.com/screencasts/installation). It's clear that he understands that different people learn in different ways: some want detailed documentation, some want tutorials, some want a visual of _how_ the library works. The screencasts also subtly counter some of the developer push-back about this approach. It's all brilliant marketing.

I knew for `django-unicorn` to be even moderately succesful it would need more than a GitHub readme with a GIF. Pretty early on I created a stand-alone [documentation site](https://www.django-unicorn.com) with an initial landing page and comprehensive documentation with example code. It is also important to have actual components that developers can interact with and see just how well they work. I'm definitely not a designer and would love help to make the documentation site better, but having a standalone site seems key to encouraging more users to try `django-unicorn`.

## Just showing up
For better or worse writing code is my day job _and_ my hobby. `django-unicorn` incrementally gets better over time because of the time I spend on it. Some weeks that might be very limited, but the average is probably 10 hours a week. Each week I slowly add new features, improve unit tests, tweak site copy, respond to GitHub issues, and improve the documentation. It only works because I enjoy all parts of the process. Like most hobbies, showing up and slowly improving a project is how to build something great.

It also helps to have friends who nudge you to create a _fix_ for a problem instead of just mindlessly complaining about it on Twitter!

Thanks for reading this far and I hope some of my lessons were useful on your journey! If you are interested in a full-stack framework for Django, please check out [https://www.django-unicorn.com](django-unicorn.com) and consider [sponsoring me on GitHub](https://github.com/sponsors/adamghill). 🦄

Cover image from photo by [De'Andre Bush](https://unsplash.com/@deandre?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText)