Depp - How it was built
Tags: go, esbuild, opensource
Published: 2021-11-03 04:54:52

---
title: Depp - How it was built
published: true
description: Fast unused and duplicate dependency checker for node modules
canonical_url: https://cryogenicplanet.tech/posts/depp#part-2
tags: go, esbuild, opensource
cover_image: https://user-images.githubusercontent.com/10355479/139758905-7f911615-84d0-46c6-805a-06f8eafaf633.png
series: Depp - A fast unused and duplicate dependency checker
---

Just briefly wanted to talk about how this built and how it works, of course the entire codebase is open-source so feel free to poke around yourself

So this package works by wrapping around the [esbuild go api](https://esbuild.github.io/api/) to add this extra functionality, so yes technically we built your whole code base to parse what dependencies you are using. This might sound incredibly stupid, but considering `esbuild` speed and flexibility, it is still significantly quicker than most solutions written in javascript

I realize this is not a great benchmark but considering most other tools don't have good monorepo support it was the best I can do. I compared `depp` (this tool) vs `depcheck` (a popular javascript tool with almost 3k stars) on a previous project [https://github.com/wei/socialify](https://github.com/wei/socialify)

```bash
depp  0.42s user 0.36s system 135% cpu 0.576 total
depcheck  1.69s user 0.13s system 116% cpu 1.567 total
```

This tool was about `2.7x` faster than `depcheck`  while practically building the entire codebase

Using `esbuild` like this also allows me to build this tool, without writing an entire `javascript/typescript` parser

## Setup

First we need to get the paths of all the `package.json` files and then all the source files,

I did this using something I built for an older project which is a modification of go standard library's `[Glob](https://github.com/CryogenicPlanet/depp/blob/master/utils.go#L79-L87)`  This modification allow me to easily find all nested `package.json` and source files

Now we can read the `package.json` files and get all the packages in the repository, we can store these as a hash map or dictionary to compare against in the future.

Finally, we can pass the source files to `esbuild` as entry points and have it do its magic

## Callbacks

First we use `Resolve` and `Load` callbacks from the `esbuild` api to check for packages as they are being imported.

There are some caveats here tho, we don't want to go down a rabbit hole of every import. Any import from `node_module` we don't care about, these are generally nested imports of modules importing other modules. So we can mark these files as `external`

Another optimization we can add is, any non `.ts, .tsx, .js,.jsx` file can be completely ignored. This is a bit painful in go because it does not support regex lookaheads (`?`) 

Now with any actual import, we can check the kind of resolve it is, whether it is a `import-statement` , `require-call` or something else we don't care about. Finally we can check if the imported path contains any of our packages

## Metafile

Additionally we use the `esbuild` meta file to do a second pass on any import we might have missed or that slip through or resolve callbacks. The `esbuild` meta file contains a bunch of information about `inputs` and their `imports` which we can use to see if a given package has been used or not

## Duplicates

While the process of finding **unused** packages is quite involved, finding duplicates is quite easy on the other hand. It is just about correctly inputting the parsed `package.json`s and checking for duplicates and storing their versions. There is a bit more work for check **unused** `@types` packages but it mostly similar to duplicates

Not going to go into it much here, that said it probably constitutes its own post of how to bundle `go` binaries with an `npm package` and how to release them

All that said this was a fairly fun project to make in the 8 or so hours it took, and it has already been fairly useful internally. Hope you find some use in the tool itself or maybe how I built it

If you want to try out the tool checkout 
{% github cryogenicplanet/depp no-readme %}