Exploring the Monorepo #1: Can't we just make project-folders?
Tags: node, typescript, productivity, monorepo
Published: 2021-06-26 18:27:30

### Table Of Contents
  * [The Good](#1)
  * [The Bad](#2)
    * [Why don't we script that?](#2-1)
    * [What about Yarn?](#2-3)

Let's get the simple solution out of the way first: Can't we just move everything into different projects within the same repository?

To test that out let's extract `web` and `api` into two separate apps, and make a `libs` folder for the shared dependencies. By moving the files around we end up with this structure:

```
webby
├── apps
│  ├── api/
│  └── web/
├── libs
│  ├── analytics/
│  ├── logging/
│  └── types/
└── tsconfig-base.json
```

And if we look at `web`'s `package.json` we see a small list of dependencies that are used entirely by `web`:

```
    "express": "^4.17.1",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-router-dom": "^5.2.0",
    "types": "file:../../libs/types"
```

> _ℹ️ As before [I've prepared this project on GitHub1s](https://github1s.com/gaggle/exploring-the-monorepo/tree/attempt-can't-we-just) if you'd like to familiarize yourself with the code. But note that now each app and lib are their own projects: You can open them up individually to avoid "drowning" in all the other code (but to do that you'll have to [clone the repository](git@github.com:gaggle/exploring-the-monorepo.git) and check out [branch `attempt-can't-we-just`](https://github.com/gaggle/exploring-the-monorepo/tree/attempt-can't-we-just) because GitHub1s doesn't support opening subfolders)._

---

## The Good <a name="1"></a>
The overview has improved greatly! The high-level architecture is now easily readable: We have two apps, and some libraries, so as a new hire I can quickly get a feel for what large-scale projects we work on.

And if we dive into `web` we see its `package.json` references the local dependency `../../libs/types`, which makes it simple to understand at-a-glance that if I work on `web` I only need to understand `libs/types` code to get my work done. How amazing!

It is worth acknowledging though that, yes, there are now more files. Where `analytics` was one file before, it is now a whole project which means it has its own `package.json`, `tsconfig.json`, + other scaffolding files. This looks quite bad with our example because our libraries are so anemic, but keep in mind we're pretending our extracted projects are those we agree are complex enough to warrant extraction. If each project actually had dozens of files and a non-trivial amount of dependencies then the high-level clarity would outweigh the added number of files. But it **is** a tradeoff: Clarity in the overview causes more project-bootstrapping files to appear. It's [simpler, not necessarily easier](https://www.johndcook.com/blog/2011/11/11/simple-versus-easy/), and only you can decide on your own balance.

## The Bad <a name="2"></a>
Unfortunately this solution **doesn't work** 😱😅. It's a frustrating conclusion because it can **appear** to work, but actually it breaks in various subtle ways

> _ℹ️ BTW, throughout this if you'd like to get back to a clean checkout state you can run this command: `git clean -dxi .`_

If we begin from a clean checkout and start the web app we immediately hit an error:

```
$ cd apps/web
$ npm ci
$ npm start
../../libs/types/src/index.ts(1,23): error TS2307: Cannot find module 'type-fest' or its corresponding type declarations.
```

What happened? This has to do with how npm installs local dependencies:

When we run `npm ci` (or `npm install`, it's the same problem either way) local dependencies are handled in a special way: A local dependency is **symlinked** into the `node_modules` folder. In this case `web` depends on `libs/types` and we can see how it's just a symlink by looking in web's `node_modules` folder:

```
$ ls -a node_modules | grep types
types -> ../../../libs/types
```

But it is **just** a symlink, npm didn't install the dependencies of `libs/types` for us like it does for normal dependencies, and so we get the `Cannot find module 'type-fest'` error because the dependency tree of `libs/types` hasn't been resolved.

Does that mean if we manually install dependencies for `libs/types` then `web` will start working?

```
$ cd ../../libs/types/
$ npm ci
$ cd ../../apps/web
$ npm start
> Started on port 3000
```

Voila! But hang on, this is a brittle and time-wasting workflow because we have to manually install each of our own dependencies… that's what npm is supposed to do for us!

### Why don't we script that? <a name="2-1">
Maybe we can script our way out of this? Here's a quick way to install all dependencies at once:

> _ℹ️ BTW I'm on macOS so I'll use its built-in tooling just to keep things simple, but of course a more mature, cross-platform solution can be made if needed._

```bash
$ cd ../..
$ for p in ./*/*; do; (cd "${p}" && npm ci > /dev/null && echo "Installed ${p}"); done
Installed ./apps/api
Installed ./apps/web
Installed ./libs/analytics
Installed ./libs/logging
Installed ./libs/types
```

And **now** everything works, right?

Mm, not quite, `web` does work but `api` doesn't:

```
$ cd apps/api
$ npm start
../../libs/analytics/src/index.ts(8,3): error TS2564: Property 'uninitializedProperty' has no initializer and is not definitely assigned in the constructor.
```

Oh boy… what's wrong now?

Well, this is a case that I've purposefully put in to mimic a real-world scenario I've seen: `libs/analytics` is not valid **strict** Typescript, it only works with the Typescript setting `strict:false`. As its own project that's fine, which can be demonstrated by running `libs/analytics`'s test-suite: 

```
$ cd ../../libs/analytics
$ npm test
Ran all test suites.
```

And if we look at its `tsconfig.json` file we see it correctly specifies the `strict:false` option:

```
$ cat tsconfig.json
  "compilerOptions": {
    "strict": false
  },
```

But `apps/api` **does** work with strict, and so specifies `strict:true` for itself, but when it runs it pulls in the analytics code via **api's** TypeScript configuration… How annoying.

I'm not sure how to fix this. Is this what Typescript references are meant for? Do I need to build each sub-project and only use the build-output? Please comment with your ideas and suggestions!

### What about Yarn? <a name="2-3"></a>
Maybe it's just npm that's the problem? Let's give Yarn a try.

First let's reset the repo and install yarn:
```
$ cd ../..
$ git clean -dxi .
$ npm install --global yarn
```

And now we can start `web`:
```
$ cd apps/web
$ yarn install
$ yarn start
> Started on port 3000
```

Hey that worked! Yarn actually fully installs local dependencies, including resolving their transient dependencies. So it avoids the "type-test" error 🎉

But this has a problem too: The dependency isn't "live", meaning changes to `libs/types` aren't reflected in `apps/web` until it re-installs its dependencies. That's not a good workflow!, we want to just change code and have it all working together, not worry about what state each project's `node_modules` folders is in. 

And besides, `apps/api` has a problem too:

```
$ cd ../api
$ yarn install
$ yarn start
SyntaxError: Cannot use import statement outside a module
```

Is there a solution for this? Some Yarn or Typescript setting to use that will make it all work?

---

It feels a lot like we're chasing down problems that I've created for myself. We can't be the first ones to tackle this problem, right? I hope I've just missed a chunk of documentation that will set us right, if you got any suggestions I'm all ears!