Rebuild only when necessary in Node
Tags: javascript, node, fsnotify
Published: 2019-06-01 00:05:38

---
title: Rebuild only when necessary in Node
published: true
description: Learn about Node's built-in filesystem watchers so that you only rebuild projects when you need to.
tags: javascript, nodejs, fsnotify
series: Blog-A-Day in June
---

If your project takes some time to prepare- maybe a compile or rewrite step- you might be waiting longer than you need to be on every load. This hits us on [Google's Santa Tracker](https://santatracker.google.com) 🎅, where we use the Closure Compiler to build most games. Closure is a great optimizer, but it's not known for speed.

So the theme of this post is: slow builds are no fun, and we're going to learn how to only run them when we need to! 🎉

## The Watch Primitive

Instead of building a game, scene or codebase every time you load a compiled resource or manually re-run a script, we can use NodeJS' `fs.watch` function to inform us whether we actually need to.

At face value, this is a straightforward method which tells you when a file has changed on-disk. Use it like this:

```js
const fs = require('fs');
fs.watch('yourfile.txt', (eventType, filename) => {
  // something happened to 'yourfile.txt': ¯\_(ツ)_/¯
});
```

This is a super efficient method because it asks your operating system to let _you_ know when something changed (not the other way around, where your program has to constantly check).

### Build Usage

Let's say you're compiling some [Less CSS](http://lesscss.com) files. You do this by compiling a file, `entrypoint.less`, that has dependencies:

```js
const less = require('less');

less.render(`@import 'entrypoint.less';`).then((output) => {
  console.info(output.css);

  // contains all files that were imported, e.g:
  //   entrypoint.less => main.less => body.less
  // will result in ['entrypoint.less', 'main.less', 'body.less']
  const files = output.imports;
});
```

Less will provide us with a simple list of files it used in the build. Some other tools might provide you with a [source map](https://developer.mozilla.org/en-US/docs/Tools/Debugger/How_to/Use_a_source_map), which also contains the names of the original files.

If __any__ of these files change, the final output is invalid, and we should rebuild it. In practice, this just means calling `fs.watch` on every file: 👀

```js
  const files = output.imports;
  files.forEach((file) => {
    fs.watch(file, () => rebuildIsNeededCallback());
  });
```

This technically works, but it doesn't really fit into a whole build system yet. Read on! 😄👍

### Caveats

While `fs.watch` is a powerful function, it has [a few caveats](https://nodejs.org/docs/latest/api/fs.html#fs_caveats). These can be summed up in a few points:

* You're not always guaranteed to be told _which_ file has changed
* On Linux, macOS and others, `fs.watch` follows the _inode_ of the watched file
  * ... if a file is _moved_, you'll be notified about changes in its new location
  * ... if a file is _replaced_, you'll be notified once, but the new file won't be automatically watched
* You need to call `.close()` on the result when you don't need it anymore—if you forget, your program will hold open listeners

In practice, these caveats means that you should use each call to `fs.watch` as a _once-off hint_ that something has changed. 💭 Think of it this way: you can't be sure exactly what changed, but it's worth checking!

Another argument in thinking of `fs.watch` as a once-off: if your dependencies _change_ by adding or removing files, it might be easier just to reset all your watchers rather than trying to keep up-to-date. 🤓

## Watch Helper

Let's put the learnings above together into a small helper that'll help you _invalidate_ code when it changes. This is what we do in Santa Tracker; we retain build output until it's no longer valid (because the underlying source has changed).

🚨 You might say "why _invalidate_, not just do a total rebuild?" Well unless you need the output as fast as possible, you're running an expensive compile step on every save.

So, the `watch` method below will accept a list of paths, watch them, and call a callback when any of them change (or a timeout passes):

```js
function watch(paths, done, timeout=0) {
  let watchers;
  let timeoutId;
  const finish = () => {
    // To finish, we close watchers (because it's not clear
    // what state they are in), cancel the timeout callback,
    // and let the user know something changed.
    watchers.forEach((w) => w.close());
    clearTimeout(timeoutId);
    done();
  };

  if (timeout > 0) {
    // If a timeout is given, 'succeed' after ~timeout. This is
    // useful to *always* rebuild or invalidate after a time.
    timeoutId = setTimeout(finish, timeout);
  }
  watchers = paths.map((p) => fs.watch(p, finish));
}
```

Be sure to take a look at the code 👆, as I've left a few comments explaining what it does. Let's put this together with our Less example from above.

## Less Is More

So just how can we invalidate output when the dependencies change?

We can do this with two methods and a cache variable:
* `getCSS` which which ensures a `Promise` containing the result is available; and
* `buildCSS` to actually rebuild when required (since this is `async`, it returns a `Promise`).

```js
let compileCache;

async function buildCSS() {
  console.debug('rebuilding CSS...');
  const output = await less.render(`@import 'entrypoint.less';`);

  watch(output.imports, () => {
    compileCache = null;  // force a rebuild next time
  }, 60 * 1000);

  return output.css;
}

// call getCSS whenever you need CSS, and it'll always be up-to-date
function getCSS() {
  if (!compileCache) {
    compileCache = buildCSS();
  }
  return compileCache;
}
```

Of course, this is a very simplified example that only caches one result: if you wanted to extend it, you'd use a dictionary of outputs, each able to be invalidated if their dependencies change.

### Finally

To finally hook up `getCSS` to the world, I'd now add a handler to your favourite NodeJS webserver so that when I load up say `/compiled.css`, it returns the result of `getCSS`, ensuring the compiled version is always up-to-date. In [Polka](https://github.com/lukeed/polka), it might look like:

```js
polka()
  .get('/compiled.css', (req, res) => {
    res.end(getCSS());
  });
```

If you're curious on more ways you might rig up a development server to do this, let me know below! 💬

## Thanks

If you're using a modern packaging system (or build tool) directly, then that tool will likely be using `fs.watch` under the hood already. Yet, I still hope you've learned something about how you can use `fs.watch` to improve your build systems!

As an aside: I've personally stopped using build tools like `gulp` and `grunt` directly in favor of custom build tools _or_ web servers that perform compilation on-demand (powered by `fs.watch`, as we do in Santa Tracker).

1 👋