DOOM! on the #emfcamp TiDAL badge
Tags: games, opensource, showdev, micropython
Published: 2022-07-22 17:40:47

---
title: DOOM! on the #emfcamp TiDAL badge
published: true
description: Porting DOOM to a Micropython badge platform
tags: #games #opensource #showdev
cover_image: https://static.tumblr.com/3673826d7ebb119208b8a0926105ed0b/uenvcsy/SV9nrd8zw/tumblr_static_82i7cdnu1p0cgcwscc8kso8s8.png
# Use a ratio of 100:42 for best results.
---
(also published on [my blog](https://www.ashbysoft.com/articles/doom-tidal/))

## Does it run DOOM?

![Angled view of emfcamp badge](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6jyxaduntnbnxv48xdwr.jpg)

It's becoming the defacto question for any piece of technology with a display on it.. fridges, printers, [all sorts of things](https://itrunsdoom.tumblr.com/) can run DOOM.

So why not this shiny new ESP32S3 based [electronic badge](https://badge.emfcamp.org/) I have just received from ElectroMagnetic Field 2022? It has a decent dual core 32bit Xtensa MCU, 8MB of RAM, 8MB of flash storage (partitioned up - see below) and an ST7789 colour pixel display.

Oh, and a joystick :)

## Similar work &amp; self-imposed constraints

There is a [port of DOOM for an ESP32](https://github.com/espressif/esp32-doom), proving that it can be done, however I have added additional constraints on myself as follows:

 * Co-exist with the Micropython environment that comes _pre-installed_ on the badge.
 * Create an app that can be distributed from the [TiDAL Hatchery](https://2022.badge.emfcamp.org/)

### So what does the micropython runtime environment look like?

 * It's based on the [Espressif IoT Development Framework (IDF)](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/get-started/index.html),
   which defines a [storage partitioning scheme](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-guides/partition-tables.html) that
   I cannot replace (but I can use to my advantage..)
 * ESP-IDF, and thus Micropython are not designed as an OS with runtime application loading/unloading in mind..
 * Micropython _is extensible at runtime_, via pre-built `.mpy` format modules, which can be compiled from `C`.

### ..and the Hatchery format?

 * A folder named after the app (eg: `/apps/Doom`), that _must_ be a python module (ie: contain an `__init__.py` file),
   plus any other files required to run.
 * The `__init__.py` must define an instance of an `App` class (or subclass) that the badge shell can locate / load.

## Cunning plan #1, and failure.

How hard can it be to simply compile up DOOM as an `.mpy` extension, patch in the I/O (display, files, user-input) back to Micropython and load it up?

It turns out the Micropython extension format is _very homebrew_ (for perfectly good reasons), and comes with significant limitations compared to an off-the-shelf application format such as `ELF` or `COFF`, in particular:

 * No declared data, there is no support for loading a `.data` segment as part of a module.
 * No static global variables (you know, the bedrock of most C programs!) as it cannot re-locate references when
   loading into memory, only _visible_ globals can be used.

I also had to choose a starting version of DOOM, with an emphasis on portability. This turned out to be the easy bit, enter [DOOM generic](https://github.com/ozkl/doomgeneric), which happily compiled and ran first time on my Linux desktop ;)

Oh, and of course I had to write a new scaler for the video to _reduce_ 320x200 => 240x135.

### The grind..

So, first remove all declared data from the DOOM sources, by moving initialisation into suitable `xx_Init()` functions if they are available, or adding them if not. ~2000 lines of code (LoC) changed, a number of ugly corner cases simply removed (like being able to change key bindings) and it runs on the desktop again - phew!

Now remove that `static` keyword from all the global variables, and prefix them all with a local module abbreviation to avoid collisions. Another ~2000 LoC changed, and it still runs on the desktop - double phew since this took several days!

Chuck all those changes into a _large_ `git patch` file, and save it away. Done.

### Tooling fail.

Compile everything according to the Micropython supplied tooling. Discover it cannot link certain object files - grrr :(

Fix [Multiple](https://github.com/micropython/micropython/issues/8781), [bugs](https://github.com/micropython/micropython/issues/8782) in [Micropython](https://github.com/micropython/micropython/issues/8783).

Chuck all the Micropython bug fixes into a `git patch` file and save that away. Done now?

### Runtime fail.

Errr nope. It turns out Micropython [makes more assumptions about how code is compiled](https://docs.micropython.org/en/latest/reference/mpyfiles.html),
in particular that all things are relocatable in memory via a single Global Offset Table (GOT). DOOM makes different assumptions, and repeatedly crashes in unpredictable locations due to memory corruption.

After a few days trying to debug this fiasco (with `printf` of course since I cannot use a debugger due to proprietary module formatting and a special loader), I give up.

## Cunning plan #2, and success (eventually!)

### Back to basics

Having dismally failed to run anything so far, I decide to start at the other end, building up from a _Hello Mum!_ program that relies only on the _built-in ESP-ROM_ (and is compiled _without_ the Micropython "special tooling" that is causing me problems), designed to run as the first user-provided code on the device, in ESP-IDF parlance, as the "secondary boot loader". Thus is born `phlashboot`, the shortest program I can write that says 'Hi' without crashing.

After some trial &amp; error with watchdog timers, this works - relief!

### A new plan forms

Remember I mentioned a `Partition table` that ESP-IDF uses? This puts application code in a specific location within flash^ (one of two 'over-the-air' or OTA update partitions), along with a header that defines _where in memory it runs, including if it's memory mapped_. The normal boot flow of an ESP-IDF application is that the secondary loader applies the defined memory mapping or copies code/data into RAM (hello `.data`!), then jumps to a defined entry point. Neat.

This is how Micropython itself get's going, can I re-use the same flow to load/run my application at _known memory locations outside Micropython_, without having to use the broken `.mpy` module format? Baby steps ensue..

^ for those of you wondering why not memory map from the filesystem part of flash? The MMU is not granular enough for this, hence having to map from a specific 'OTA' partition offset.

### Gently does it..

I adapt `phlashboot` so it doesn't expect to be the bootloader, instead it looks like an OTA application, which the ESP-IDF secondary loader is happy to memory map (mmap) and run, next step..

After a great deal of reading through [technical manuals](https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/hw-reference/index.html) and [rummaging in the code](https://github.com/espressif/esp-idf), creating the `romread` test application and generally poking at (and crashing) my device, I understand how the memory manager works, what I can and cannot touch and in theory how I can _memory map and execute another app within micropython_. There is also a fair amount of investigation into the Micropython memory layout, see `memstuff` folder! Thus is born `doomloader.mpy` a short and _separate_ Micropython module that pokes at the MMU to load an OTA app into Micropython-safe address space. It works :D, next step..

### DOOM is back baby!

I can now use standard tooling, to build a standard application and run it on device "underneath" Micropython, so back to DOOM. There is some fiddling with compiler options and load addresses to get a sensible OTA application out, which fits easily into a flash (~460k, OTA partitions are 2MiB), the existing plan for connecting in I/O is still intact(!) so in theory it should work... and it doesn't.

It turns out that DOOM expects a little more POSIX compliance from it's C library than Micropython provides, in particular the `printf` implementation is lacking many features. The good part is that because I'm building an entirely separate binary application, I can [_use someone elses printf_](https://github.com/charlesnicholson/nanoprintf), and finally, after a _lot_ of pain, it runs!

The colours are awful. I can't actually control anything yet. I'm deliriously happy :D

### Final fixes

The colour being mangled turns out to be an endian difference between the MPU and the display device, easily determined by looking at the [code for the display driver](https://github.com/russhughes/st7789_mpy), which byte-swaps everything when drawing into the device, unless it's blitting a buffer, when it _assumes it's already swapped_. Fixed.

Providing user input is a case of finding the right DOOM keys to fake when pressing joystick buttons or others on the badge.

## Show me the code!

To horribly mis-quote another of my favourite games [come get some baby!](https://github.com/phlash/doom-tidal)

..OR..

Grab the built objects [from the hatchery](https://2022.badge.emfcamp.org/projects/doom)

## The required demo video..

<video width="960" height="540" src="https://www.ashbysoft.com/images/TiDAL-DOOM.mkv" controls >Sorry, your browser doesn't support HTML5 video</video>