Better Backend DX: Fastify + ESBuild = ⚡️
Tags: node, webdev, productivity, tutorial
Published: 2022-05-09 14:50:39

Hi! I'm David Peng👋, you can find me on Twitter: [@davipon](https://twitter.com/davipon).

I'm currently working on an internal legacy project which I mentioned in my old blog post: "[Supercharge Web DX in Svelte Way](https://dev.to/davipon/supercharge-web-dx-in-svelte-way-3nke)".

I built a back-end service from scratch in the past few months, part of the monolithic to microservice migration. It was also my first time making all technical decisions like:

- Which framework to choose? Express, Koa, Fastify, or NestJS? 
- DB choice? SQL or NoSQL? Use ORM/ ODM or not?
- Understand networking like transport protocols (UDP, TCP, and TLS), application protocols (HTTP/1.1, HTTP/2)
- Do we need a load balancer/ reverse proxy? Apache, Nginx, HA Proxy, or Envoy?
- Containerize the application or not? Will we need orchestration in the future?
- How to build & test in the development, stage, and production environment?
- What about the CI/ CD pipeline? DevSecOps?
- 🤯
![mind blown](https://media.giphy.com/media/3o8dFn5CXJlCV9ZEsg/giphy.gif)


It's intimidating😅, but I learned so much by getting my hands dirty. Even though we haven't finished the migration yet, I still want to write them down and start a new blog series.

I'll share my experience and thought process of building a production-ready backend service with __a better DX__.

There will be four parts (I might change the title in the future 🙃):
1. Better Backend DX using Fastify & ESBuild
2. Build a REST API using MongoDB
3. Build a minimal docker image
4. Add Envoy as a sidecar proxy

Let's start with the first one!

## Better Backend DX (Developer Experience): Fastify + ESBuild = ⚡️

Below are my two beliefs about solving a real problem:

> Build things that just work.

> Better DX leads to better UX and better products.

When building a SvelteKit app, it's full of joy with a great DX. (Kudos👏 to the community and modern toolings.) But when it comes to JavaScript backend development, the DX is still like in the stone age.

## How is the general backend DX?

Just my two cents:

- Lacks modern toolings like hot module replacement
- Restrictions of Node.js - immature ESM support leads to extra setup or compilation to `cjs`
- `tsc`/ `ts-node` + `nodemon` still slow as hell
- No interest in bundling production code (hard to debug? we don't care because it doesn't ship to clients? 🤷)

I couldn't stop thinking about these when I started to develop a pure backend. It seems to me that there is a massive gap between the frontend & backend world in terms of toolings and DX.

Let's say you want to build a node app using TypeScript, and you'd probably use `tsc`, `ts-node`, `concurrently`, or `nodemon` to:
1. Watch and compile your project from `ts` to `js`
2. Start the server

It works, but the problem is that you'll have a broken/ long feedback loop. It recompiles **the whole project** whenever a file changes, and `tsc` is way too slow:

![benchmark](https://datastation.multiprocess.io/react-benchmark-large.png)

ref: [Benchmarking esbuild, swc, tsc, and babel for React/JSX projects](https://datastation.multiprocess.io/blog/2021-11-13-benchmarking-esbuild-swc-typescript-babel.html)

> That's how I feel! I missed Hot Module Replacement and Vite goodness... wait, can we use it?

I used Vite in my React project, and SvelteKit also uses Vite under the hood. So I wondered if it's possible to use Vite in the backend development?

## Vite for Node apps?

Yes, there is a Vite plugin: [`vite-plugin-node`](https://github.com/axe-me/vite-plugin-node) which leverages Vite's SSR mode to bring HMR to the Node dev server and also provides adapters for Express, Koa, and Fastify.

I also found a nice [vite-fastify-boilerplate](https://github.com/ManUtopiK/vite-fastify-boilerplate) which uses the plugin & [Vitest](https://vitest.dev).

After trying them out, the feedback loop was much shorter than before thanks to HMR & `esbuild`, but there are two problems:

1. [Websockets don't seem to work, at least with Fastify #22](https://github.com/axe-me/vite-plugin-node/issues/22)
2. I Haven't found a way to **bundle** production code using `vite-plugin-node`. ([Vite SSR mode doesn't support bundling built-in Node.js lib](https://vitejs.dev/guide/ssr.html#ssr-bundle)

> I like to bundle/ minify the production code because I want to build a docker image that doesn't need to copy the huge `node_modules` to the final image. I'll explain in a future post 😉.

Let's step back and compare different approaches:

||`tsc`+`nodemon`|`vite-plugin-node`|
|---|---|---|
|HMR|Recompile the entire app every time.|Yes|
|Compiler|`tsc`|`esbuild` or `swc`|
|Problem|Too slow|WebSockets issue, bundling options|

Hmm🧐, it seems I need to find an alternative approach, so I started to experiment with ESBuild.

## Use [esbuild](https://esbuild.github.io) as a compiler and backend bundler

![esbuild](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lc3j4951kamef3h1euf9.png)

If we don't use Vite, we can't have HMR in the dev server.
But we can still use `esbuild`, the compiler that Vite uses, to replace `tsc`.

`esbuild` is so fast that even if we use `nodemon` to monitor and recompile the entire app, we can still have a short feedback loop.

> [`esbuild-node-tsc`](https://github.com/a7ul/esbuild-node-tsc#live-reloading-nodemon-%EF%B8%8F-esbuild-node-tsc) is one of the packages that use this approach.

> You can also try `ts-node` + `swc` + `nodemon`. ref: https://typestrong.org/ts-node/docs/swc

To have a better backend DX, there are more factors to consider besides dev server and production bundling. Before we start to set up the `fastiy` + `esbuild` project, let's talk about Fastify.

## Why did I choose [Fastify](https://www.fastify.io)?

![fastify](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nr4wu36kr8tq24x70jjp.png)

Here are the reasons:

- Support TypeScript out of the box
- Baked-in validation using JSON-Schema
- Extensibility - hooks, plugins, and decorators
- **Good DX**, e.g., great logger using `pino`, rich plugin ecosystem.
- @fastify/autoload enables filesystem-based routes & plugins

> Performance was not my primary consideration. 

> Both Express & Koa are fast enough for a small project. IMHO, It would be better to spend dev time optimizing and parallelizing backend calls and DB queries than speeding up a router by 2ms.![shrugging](https://media.giphy.com/media/1X8TXeLwY2ojkQgrOo/giphy.gif)

## Let's build a simple `fastify` + `esbuild` app!

>This section aims to build a scalable starter template with a better DX.

> Here is the GitHub repo: [github.com/davipon/fastify-esbuild](https://github.com/davipon/fastify-esbuild)

```zsh
# Create a new project
mkdir fastify-esbuild
cd fastify-esbuild
pnpm init
# Install dependencies
pnpm add fastify fastify-plugin @fastify/autoload
# Install devDependencies
pnpm add -D typescript @types/node nodemon esbuild
```

Create a `src` folder and `index.ts` under `src`:
```ts
// src/index.ts
import Fastify from 'fastify'

const fastify = Fastify({
  logger: true
})
// Declare a route
fastify.get('/', function (request, reply) {
  reply.send({ hello: 'world' })
})
// Start the server
const start = async () => {
  try {
    await fastify.listen(3000)
  } catch (err) {
    fastify.log.error(err)
    process.exit(1)
  }
}
start()
```

We use `nodemon` to monitor & restart the dev server.
Let's create `nodemon.json` under your project root folder:
```json
// nodemon.json
{
  "watch": ["src"],
  "ignore": ["src/**/*.test.ts"],
  "ext": "ts,mjs,js,json,graphql",
  "exec": "esbuild `find src \\( -name '*.ts' \\)` --platform=node --outdir=dist --format=cjs && node dist",
  "legacyWatch": true
}
```
The `"exec"` script does following:
1. Compile all `ts` files under `src` folder to `cjs` (CommonJS) format and output to the folder `dist`
2. Start the dev server with `node dist` (or `node dist/index.js`)

> **Caveat**

> `find src \\( -name '*.ts' \\)` is only available in Linux/ macOS.

> Will provide an alternative solution for Windows later. (ref: https://github.com/evanw/esbuild/issues/976)

> The reason to assign all files but not just the entry like `src/index.ts` in the esbuild script is that we need to use `@fastify/autoload` later.


Then add scripts in `package.json`:
```json
...
"scripts": {
    "dev": "nodemon",
    "build": "rm -rf build && esbuild `find src \\( -name '*.ts' \\)` --platform=node --outdir=build --bundle",
    "start": "pnpm build && node build"
  },
```
Let's try `pnpm dev`. You should see something like this:

![pnpm dev](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l91dhj1vyr82nbk0gjjk.png)

### Enables filesystem-based routes & plugins using [`@fastify/autoload`](https://github.com/fastify/fastify-autoload)

First create a `routes` folder under `src`, and then create a `root.ts` under `routes`:
```ts
// src/routes/root.ts
import { FastifyPluginAsync } from 'fastify'

const root: FastifyPluginAsync = async (fastify, opts): Promise<void> => {
  fastify.get('/', async function (request, reply) {
    return { root: true }
  })
}

export default root
```
Then in your `src/index.ts`:
```ts
import Fastify from 'fastify'
import { join } from 'path'
import autoLoad from '@fastify/autoload'

const fastify = Fastify({
  logger: true
})

// Will load all routes under src/routes
fastify.register(autoLoad, {
  dir: join(__dirname, 'routes')
})

const start = async () => {
  try {
    await fastify.listen(3000)
  } catch (err) {
    fastify.log.error(err)
    process.exit(1)
  }
}
start()
```

Let's give it a try!

![fastify-autoload demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cyzruxbxt444liy60yh2.gif)

Awesome🥳! Now let's add more routes:

Create a `examples` folder under `routes` and create `index.ts` in it:
```ts
// src/routes/examples/index.ts
import { FastifyPluginAsync } from 'fastify'

const examples: FastifyPluginAsync = async (fastify, opts): Promise<void> => {
  fastify.get('/', async function (request, reply) {
    return 'Hi there!'
  })
}

export default examples
```
Here is the demo:

![add-routes-demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dlqqzwqwnedodizks3kp.gif)

![add-routes-demo2](https://cdn.hashnode.com/res/hashnode/image/upload/v1652003693792/xWvly-0jf.gif)

With `@fastify/autoload`, we can easily add plugins and routes. In the future, you might want to deploy some of those independently. It also provides an easy path to a microservice architecture.

You can see a basic structure with routes and plugins like this:

![folder structure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7lstmn72e0tg4e6qdnai.png)

### ⚠️ Windows' solution for the `esbuild` script

Create `esbuild.js` under the project root folder.
```js
const env = process.argv[2]
/* eslint-disable @typescript-eslint/no-var-requires */
/* eslint-disable no-undef */
const fs = require('fs')
const path = require('path')
const esbuild = require('esbuild')

let fileArray = []
const getFilesRecursively = (dir) => {
  const files = fs.readdirSync(dir)
  files.forEach((file) => {
    const filePath = path.join(dir, file)
    if (fs.statSync(filePath).isDirectory()) {
      getFilesRecursively(filePath)
    } else {
      fileArray.push(filePath)
    }
  })
}
getFilesRecursively('src')

const entryPoints = fileArray.filter((file) => file.endsWith('.ts'))

esbuild.build({
  entryPoints,
  logLevel: 'info',
  outdir: env === 'dev' ? 'dist' : 'build',
  bundle: env === 'dev' ? false : true,
  platform: 'node',
  format: 'cjs'
})

```
Replace `esbuild` script in both `nodemon.json` and `package.json`:
```json
// nodemon.json
{
  ...
  "exec": "node esbuild.js dev && node dist",
}
```
```json
// package.json
{
  ...
  "script": {
    "build": "rm -rf build && node esbuild.js",
  }
}
```

## That's it! 🎉 This is Part 1. 

Thank you for your reading!

You may notice that this `fastify` + `esbuild` is heavily inspired by modern web frameworks and toolings. 

What motivated me to write this series was the frustration. You can find many resources and articles about modern tools and how they improve frontend DX, but only a few in the backend world talk about it. 

In the frontend world, **Vite**'s approach to leveraging ESM & HMR to tackle slow server start and bundling is fantastic. I hope we can have native backend/ Node.js support in the future.

In the next blog post, I'll share how we can use this `fastify` + `esbuild` template to:
- Build REST API using MongoDB
- Use `json-schema-to-ts` to validate data and stop typing twice
- Use native MongoDB driver instead of ODM like Mongoose

Kindly leave your comment and thoughts below!

See you in the next one!
![let's go](https://media.giphy.com/media/cdNSp4L5vCU7aQrYnV/giphy.gif)