Monkey-patching the Express router
Tags: node
Published: 2021-07-08 22:55:03

I maintain [Scribe](http://scribe.knuckles.wtf), a library for automatically generating your HTTP API documentation from your codebase. When you run the `generate` command, Scribe fetches your routes from your codebase, uses a bunch of strategies to figure out details about them, and transforms those details into a HTML docs page. There are a lot of moving parts and complexity in this process, but this post is about the first part: how Scribe is able to fetch the routes from your app.

In the Laravel version, this process is [hella easy](https://github.com/knuckleswtf/scribe/blob/4fad273447b548c13f899948b4a44e50f5d28590/src/Matching/RouteMatcher.php#L46): just call `Illuminate\Support\Facades\Route::getRoutes()`, and you get an array of `Route` objects, with each object containing the route details, including the path, HTTP methods, and handler \(controller/method\). It's a similar thing for [the AdonisJS version](https://github.com/knuckleswtf/scribe-js/blob/d6c594dbe46bbf48c8fda0fe09c8d5edb5cff23d/frameworks/adonis/commands/GenerateDocumentation.js#L65).

The Express version is another story, though. Express doesn't provide a simple method you can call on the `app` to fetch all defined routes. This means it's time for a hobby of mine: monkey\-patching. When our `generate` command is run, we'll dynamically modify the Express code so that it gives us what we want.

## First attempt: decorating

My first attempt had a couple of moving parts:

* a ["decorator" function](https://github.com/knuckleswtf/scribe-js/blob/fae090cdc953d96537d4379cf5b9b4162f475d9d/frameworks/express/src/decorator.js#L11) that would take the Express app object and modify its methods \(`app.get(...)`, `app.post(...)`, etc\) so that they recorded _where_ they were called. This part was important so I could fetch any docblocks there later on. For the decorator to be useful, it had to be called before the user started registering routes.
* a [`getRoutes` function](https://github.com/knuckleswtf/scribe-js/blob/fae090cdc953d96537d4379cf5b9b4162f475d9d/frameworks/express/src/get_routes.js) that would receive the `app` object and try to fetch the user's routes from the Express "API".

Here's what the end user's code would look like:

```js
// app.js

const express = require('express');
const app = express();

// Add this 👇 (the decorator)
require('@knuckleswtf/scribe-express')(app); 

// Register your routes...
app.get(...);

// Add this 👇 (needed for getRoutes)
module.exports = app;
```

The user would run the Scribe command like this: `npx scribe generate --app app.js`, and it basically did this:

```js
const appFile = args.app;
const app = require(appFile); // (1)
const routes = getRoutes(app); // (2)
```

The line \(1\) would execute the app file, thereby running the decorator and registering the routes. Then the next line would pass the exported `app` object to our `getRoutes` function, which would try to fetch the routes from it.

## Assessment

There are three main metrics I use to judge the success of a process like this: how easy it is for a user to opt in, how easy it is to implement, and how reliable it is. How did this perform?

### User integration: not great

There were too many steps and things to remember:

* The user had to remember to add `require('@knuckleswtf/scribe-express')(app)`.
* They had to remember to add that line _after_ creating the app, but _before_ registering any routes.
* They had to make sure they exported the `app` object from the file \(many people don't, because it's not needed for anything\).
* On top of all this, they had to run the `generate` command.

That's too many points of failure for me; too many opportunities for users to make mistakes. It's not awful, but not ideal. In an ideal monkey\-patch, the user should only have to make a very small modification to their code, or none at all.

### Ease of implementation: poor

I used quotes earlier when talking about Express' API, because Express doesn't have one. Its internals are a complicated mess. It works, and it does many clever things, but it's unnecessarily complex. You have all sorts of objects all over the place—routers, layers, stacks, routes, handles, and whatnot. Routes can have layers, and layers can have routes, and a layer can be a route, or some other shit. And mixed right in there with routes are middleware like authentication, so you need to filter those out too. Express isn't made for developers to hook into that way.

Figuring out how to extract the actual routes from the tangle of layers and stacks took several days. And by the time I was done, I never wanted to touch the thing again.

### Reliability: poor

On the reliability front, this was also pretty poor. Because Express' internals were so complicated to figure out, a lot of what I did was based on assumptions, guesses and workarounds.

For instance, Express doesn't retain the original path you set when you create a route \(like `app.post('/path', handler)`\). Instead, it converts the path to a regex, like `/^\/?path\/?$/`. This means that, when fetching the routes, I had to [convert the regexes back to strings](https://github.com/knuckleswtf/scribe-js/blob/fae090cdc953d96537d4379cf5b9b4162f475d9d/frameworks/express/src/get_routes.js#L81), which works, but I don't know if it's guaranteed to always give exactly what the user wrote.

There was also the fact that people can register routes in Express in a bunch of different ways—you can have multiple apps, routers, sub\-routers, sub\-apps, and so on, and Express doesn't combine them into one simple data structure. Since my decorator implementation was tied to a specific `app` instance, it made these other options really difficult to get right.

_Seriously, this experience was the final thing that convinced me people need to stop using Express. It was a key part of Node.js' development, but I don't buy this middleware approach that leaves you having to implement everything yourself, and doesn't expose a proper API._ 😕

## A better solution

My first attempt was more of "decorating" than monkey\-patching—it only added some specific behaviour to the `app` object that was passed to it. But it would be better if we had true monkey\-patching: attaching this behaviour automatically to any instance of `express()`, even those created in a different file. There's good news: in Node.js, there are defined APIs to do that, by hooking into the module system.

For instance, if I overwrite`require.cache.express` \([docs](https://nodejs.org/api/modules.html#modules_require_cache)\), when the user calls `require('express')`, they'll get my new version. This means I can give them a modified `express`, so that when they call `express()`, the `app` object they get is already decorated. No need to pass in your `app` manually to the decorator anymore.

This led to another realization: _since users don't need to pass in an `app` anymore, then they don't even need to call the decorator in their code at all_. The `generate` command could call the decorator itself to monkey\-patch Express, before executing the user's app file. So now we can eliminate the first two frustrations I had with the user integration process.

The final big improvement came from me realising that I could skip the Express "API" altogether. When a user calls `app.get('/path', handler)` on our monkey\-patched app instance, we already have all the information we need — the path, the handler, the HTTP method, the file name and line. So we can just record this route somewhere before handing over to Express to handle it, rather than trying to fetch it from the router later.

This was awesome\! It eliminated my final concerns: this approach was way more reliable, since I wasn't relying on Express' internal API \(layers and stacks\), but their public API \(`app.get(...)`\), which was documented and stable. I could get the original path, handle sub\-apps and sub\-routers, and users didn't even need to export the `app` object anymore.

## Implementation

I was going to implement this manually, but then I remembered my friends at Elastic [specialise in this](https://blog.shalvah.me/posts/a-shallow-dive-into-auto-instrumenting-node-js-applications-with-elastic-apm). Thanks to their work with `require-in-the-middle` and `shimmer`, it was fairly straightforward, although it still took some work to figure out _what_ exactly I needed to patch. For Express, there were three main entry points:

1. the HTTP methods on `express.Route.prototype`, because Express internally creates a new instance of `Route` and calls `route.get(...)` when you call `app.get(...)` or `router.get(...)`.
2. the `express.application.use()` and `express.Router.use()` methods \(for `app.use('path', router)`\).
3. the `express.Router.route()` method, which is called when you create a route using `app.route('path').get(handler)`. Here we just record the router the route belongs to, so that when our patched `get` is called \(number 1 above\), we have all the details we need.

You can see the full code [on GitHub](https://github.com/knuckleswtf/scribe-js/blob/f85802aced8bd8ffddb121fdc1f57b4d13a71573/frameworks/express/src/decorator.js), but the decorator now looks like this:

```js
function decorator() {
    decorateExpress();
}

// A flag that keeps us from doing the monkey-patch multiple times
decorator.decorated = false;

// The place where we record our routes as they're registered
decorator.subRouters = new Map();

function decorateExpress() {
    const hook = require('require-in-the-middle');
    const shimmer = require('shimmer');

    // Hook into require('express'') and return ours
    hook(['express'], function (exports, name, basedir) {
        if (decorator.decorated) {
            return exports;
        }

        // Handle app.get(...), app.post(...), and so on
        const httpMethods = ['get', 'post', 'put', 'patch', 'head', 'delete'];
        httpMethods.forEach(function shimHttpMethod(httpMethod) {
            shimmer.wrap(exports.Route.prototype, httpMethod, original => {
                return patchHttpVerbMethod(original, httpMethod);
            });
        });

        // Handle sub-routers and sub-apps 
        // ie app.use('/path', otherApp), app.use('/path', otherRouter), router.use('/path', otherRouter), etc
        shimmer.wrap(exports.application, 'use', patchAppUseMethod);
        shimmer.wrap(exports.Router, 'use', patchRouterUseMethod);

        // Handle app.route(path).get()
        shimmer.wrap(exports.Router, 'route', original => {
            return function (...args) {
                const routeObject = original.apply(this, args);
                // Track the router that this route belongs to
                routeObject.___router = this;
                return routeObject;
            }
        });

        decorator.decorated = true;
        return exports;
    });
}
```

The `patchHttpVerbMethod`, `patchAppUseMethod`, and `patchRouterUseMethod` do the real work of recording the routes in `decorator.subRouters`

The`decorator.subRouters` map is the key that allows us to support sub\-apps and sub\-routers. It acts as a sort of tree. Every route in Express is added to a router instance. A user can have multiple apps and routers, and mount them on each other using `app.use()` or `router.use()`. But since we won't know _when_ they'll call `app.use('/somepath', thisrouter)`, we store all the routes they add on `thisrouter` in `decorator.subRouters[thisrouter]`. When the user finally calls `app.use('/somepath', thisrouter)`, we fetch the routes from `decorator.subRouters[thisrouter]` and add them to the `decorator.subRouters[app]`.

```js
const app = express();

// decorator.subRouters is empty.

app.get('/a', someHandlerFn);

// decorator.subRouters now looks like this:
//       app =>  GET /a

app.post('/a', someHandlerFn);

// decorator.subRouters now looks like this:
//       app => GET /a
//              POST /a

const subRouter = express.Router();
subRouter.get('/a', someHandlerFn);

// decorator.subRouters now looks like this:
//       app       => GET /a
//                    POST /a
//       subRouter => GET /a

app.use('/b', subRouter);

// decorator.subRouters now looks like this:
//       app => GET /a
//              POST /a
//              GET /b/a
```

By the time all the routes have been registered, there should be only one router left in `decorator.subRouters`—the main router. To fetch the routes, the `generate` command \([GitHub](https://github.com/knuckleswtf/scribe-js/blob/f85802aced8bd8ffddb121fdc1f57b4d13a71573/frameworks/express/src/cli/generate.js)\) simply has to fetch those routes.

```js
// Monkey-patch the Express router so that when a user adds a route, we record it
require('./decorator');

// Execute app file so routes are registered
require(appFile);

const [routes] =  [...decorator.subRouters.values()];
```

And that's it. The new solution is much better in terms of user experience, and was much easier to implement, although still tasking. There are still a few parts that are fragile, where I had to rely on some specific internal detail of the Express code \(like [this](https://github.com/knuckleswtf/scribe-js/blob/f85802aced8bd8ffddb121fdc1f57b4d13a71573/frameworks/express/src/decorator.js#L95)\), but it's way more reliable than before, plus there are now tests to check that it works. So it's a definite win.