Recursive Directory Removal in Node.js
Tags: node, javascript
Published: 2019-09-09 17:24:53

---
title: Recursive Directory Removal in Node.js
published: true
tags: node,javascript
canonical_url: https://boneskull.com/recursive-directory-removal-in-node-js/
---

![Recursive Directory Removal in Node.js](https://boneskull.com/content/images/2019/09/Sierpinsky_triangle_-evolution-.png)

Recursive directory removal [has landed](https://github.com/nodejs/node/pull/29168) in Node.js v12.10.0!

This has been a long-standing feature request.  New Node.js developers often express incredulity  when they discover this particular “battery” isn’t included in Node.js.

Over the years, userland modules ([**rimraf**](https://npm.im/rimraf), [**rmdir**](https://npm.im/rmdir), [**del**](https://npm.im/del), [**fs-extra**](https://npm.im/fs-extra), etc.) have heroically provided what core did not. Thanks to the superbad maintainers-of and contributors-to these packages!

Here's a little story about how it came to pass—and why something _so seemingly simple_ as `rm -rf` isn’t necessarily so.

## About Node.js’ Filesystem Operations

First, I want to explain a bit about how Node.js works under the hood with regards to filesystem operations.

[**libuv**](https://libuv.org) provides filesystem operations to Node.js.  Node.js’ `fs` module is just a JavaScript file which provides the `fs.*` APIs; those APIs call into an internal C++ binding (you could think of this as a “native module”). That binding is _glue_ between **libuv** and the JavaScript engine ( **V8** ).

Here’s an example.  At the lowest level, **libuv** provides a C API (`uv_fs_rmdir`) to make the a system call to remove a directory.

```js
const fs = require('fs');

// `rmdir` is just a function which calls into a C++ binding.
// The binding asks libuv to remove the "/tmp/foo" directory.
// Once libuv returns a result, the binding calls `callback`
fs.rmdir('/tmp/foo', function callback(err) {
  if (err) {
    // handle error
  }
});
```

Importantly, Node.js makes only a _single call_ to **libuv** above_._

In fact, until recently, Node.js’ `fs` bindings follow a pattern: single calls into **libuv**. `fs.readFile`, `fs.stat`, `fs.unlink`; these are all just _one_ call.

_Oh_, that recent change? It was [recursive `fs.mkdir`](https://nodejs.org/api/fs.html#fs_fs_mkdir_path_options_callback).  I’ll explain what makes it different.

## Shell Operations vs. System Operations

Developers may not think about this much because it’s so well-abstracted by our tools.  Take `mkdir`, for example:

```bash
$ mkdir ./foo
```

`mkdir` is a command-line utility (which flavor, exactly, depends on your operating system).  It’s _not_ a system call.  The above command may only _execute_ a single system call, but the following may execute several:

```bash
# creates dirs foo, then bar, then baz, ignoring dirs that already exist
$ mkdir -p ./foo/bar/baz
```

Unless our tools have _transactional_ behavior—they can “commit” or “roll back” operations—it’s possible for this command to _partially_ succeed (though maybe not obvious in this case, but trust me).

What happens if `mkdir -p` fails halfway through?  _It depends._ You get zero or more new directories.  Yikes!

If that seems weird, consider that the user may _want_ to keep the directories it _did_ create.  It’s tough to make assumptions about this sort of thing; cleanup is best left to the user, who can deal with the result as they see fit.

How does this relate to Node.js?  When a developer supplies the `recursive: true` option to `fs.mkdir`, Node.js will potentially ask **libuv** to make _several_ system calls—_all, some, or none_ of which may succeed.

Previous to the addition of recursive `fs.mkdir`, Node.js had no precedent for this behavior.  Still, its implementation is relatively straightforward; when creating directories, the operations must happen both _in order_ and  _sequentially_—we can’t create `bar/baz/` before we create `bar/`!

It may be surprising, then, that a recursive `rmdir` implementation is another beast entirely.

## There Was An Attempt

I was likely not the first to attempt to implement a recursive `rmdir` in Node.js at the C++ level, but I _did_ try, and I’ll explain why it didn’t work.

The idea was that a C++ implementation could be more performant than a JavaScript implementation—that’s probably true!

Using `mkdir` as a template, I began coding.  My algorithm would perform a depth-first traversal of the directory tree using **libuv** ’s `uv_fs_readdir`; when it found no more directories to descend into, it would call `uv_fs_unlink` on each file therein. Once the directory was clear of files, it would ascend to the parent, and finally remove the now-empty directory.

It worked!  I was very proud of myself.  Then I decided to run some benchmarks against [**rimraf**](https://npm.im/rimraf). Maybe I shouldn't have!

I found out that my implementation was faster for a very small _N_, where _N_ is the number of files and directories to remove.  But _N_ didn’t have to grow very large for userland's **rimraf** to overtake my implementation.

Why was mine slower?  Besides using an unoptimized algorithm, I used recursive `mkdir` as a template, and `mkdir` works _in serial_ (as I mentioned above).  So, my algorithm only removed _one file_ at a time.   **rimraf** , on the other hand, queued up many calls to `fs.unlink` and `fs.rmdir`.  Because **libuv** has a thread pool for filesystem operations, it could speedily blast a directory full of files, only limited by its number of threads!

> Note that when we say Node.js is single-threaded, we're talking about the _programming model_.  Under the hood, I/O operations are multithreaded.

At this point, I realized that if it was going to be “worth it” to implement at the C++ layer—meaning a significant performance advantage which outweighs-the- maintenance-costs-of-more-C++-code—I’d have to rewrite the implementation to manage its _own_ thread pool.  Of course, there’s no great precedent for _that_ in Node.js either.  It’d be possible, but very tricky, and best left to somebody with a better handle on C++ and multithreaded programming.

I went back to the [Node.js tooling group](https://github.com/nodejs/tooling) and explained the situation.  We decided that the most feasible way forward would be a pure-JavaScript implementation of recursive directory removal.

## Let’s Write It In JavaScript!

Well, that was the idea, but we didn’t get very far.  We took a look at the [source of **rimraf**](https://github.com/isaacs/rimraf/blob/master/rimraf.js), which is the most popular userland implementation. It’s not as straightforward as you’d expect!  It covers many edge cases and peculiarities (and all of those hacks would need to be present in a Node.js core implementation; it needs to work like a consumer would expect).

Furthermore, **rimraf** is stable, and these workarounds have proven themselves to be robust over the years that it’s been consumed by the ecosystem.

I won’t attempt to explain what **rimraf** must do to achieve decent performance in a portable manner—but rest assured it’s sufficiently _non-trivial_.  _So_ non-trivial, in fact, that it made more sense to just _pull **rimraf** into Node.js core_ instead of trying to code it again from scratch.

So that’s what we did.

## It’s Just rimraf

[Ian Sutherland](https://github.com/iansu) extracted the needed code from **rimraf**.  In particular, **rimraf** supplies a command-line interface, and we didn’t need that.  For simplicity (and to eliminate dependencies) glob support (e.g., `foo/**/*.js`) was also dropped (though it [may still have a future](https://github.com/nodejs/tooling/issues/38)).  After this, it was a matter of integrating it into a Node.js-style API, and the needed docs and tests.

To be clear, recursive directory removal in Node.js does _not_ make rimraf obsolete. It _does_ mean that for many use cases, Node.js’ `fs.rmdir` can get the job done.  Stick with **rimraf** if you need globs or a portable command-line utility.

Thanks to [Isaac Schlueter](https://github.com/isaacs/) for **rimraf** —and to bless Node.js’ copy-and-paste efforts.

## In Conclusion

That’s the story of Node.js’ recursive `rmdir` thus far. Want to help write the rest? Come participate in the [Node.js Tooling Group](https://github.com/nodejs/tooling), where we’re looking to make Node.js _the best platform it can be_ for building CLI apps.

> _Acknowledgements to Isaac Schlueter and Ian Sutherland for reviewing this post._

> _This post originally appeared on [boneskull.com](https://boneskull.com/recursive-directory-removal-in-node-js/) on Sep 9, 2019._