A little npm head-scratcher
Tags: bash, cli, zsh, npm
Published: 2020-11-07 07:45:42

The other day I ran into an interesting puzzle, which was fun to sort out, and I learned enough in the process that I thought I should write it down.

## The setup

A [JavaScript project I maintain](https://github.com/danburzo/percollate) has the following file structure, abridged:

```
src/
  util/
     slurp.js
  cli-opts.js
test/
  cli-opts.test.js
index.js
```

Two tools I normally use for JavaScript projects are `tape` for tests and `eslint` for linting, and these I ultimately ran as:

```bash
tape test/**/*.test.js
eslint {src,test}/**/*.js *.js
```

...but they're actually stored as npm scripts in my `package.json` file:

```json
{
	"scripts": {
	  "test": "tape test/**/*.test.js",
	  "lint": "eslint {src,test}/**/*.js *.js"
	}
}
```

...and then invoked with `npm run test` and `npm run lint` respectively.

These commands worked out great. So great that I added them to a pre-commit hook, and ran them in GitHub Actions, to make sure the code is top-shelf quality.

Then one code change, which passed the pre-commit hooks, suddenly blew up the GitHub action: `npm run lint` had found two linting errors in the `src/cli-opts.js` file.

Huh. I fire up my terminal, on which I've been running the zsh shell for the last few years, and execute `npm run lint`, as one does.

No errors. What's going on?

In the off-chance you want to figure it out yourself (which I imagine is no fun without the actual setup with which to play around), stop here before reading the explanation below.

## The explanation

What's going on is shells.

Remember I casually mentioned I run zsh?  It turns out this matters. Bringing up the [manual for `npm run`](https://docs.npmjs.com/cli/v6/commands/npm-run-script) revealed this piece of hitherto unknown information:

> The actual shell your script is run within is platform dependent. By default, on Unix-like systems it is the `/bin/sh` command, on Windows it is the `cmd.exe`. The actual shell referred to by `/bin/sh` also depends on the system.

Wait a second, you mean to tell me I've been running npm scripts with `GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)` all along?!

One aspect for which the choice of shell is important is that different shells have different [glob](https://en.wikipedia.org/wiki/Glob_%28programming%29) expansion rules.

zsh supports [recursive expansion](http://zsh.sourceforge.net/Doc/Release/Expansion.html#Recursive-Globbing) of the  `**/` pattern, so that `src/**/*.js` matches both `src/util/slurp.js` and `src/cli-opts.js` while bash 3.2 only matches the former. It's only in version 4 that [bash gets a `globstar` option](https://www.linuxjournal.com/content/globstar-new-bash-globbing-option) that allows recursive expansion, and even that might not be enabled by default.

Furthermore, when a glob pattern has no matching files, zsh throws an error. By default, bash outputs the glob pattern unchanged.

To see what the problem might be with these two pairs of diverging behaviors, here are the file structure and the glob patterns again, along with their expansion in zsh and bash:

```bash
src/
  util/
     slurp.js
  cli-opts.js
test/
  cli-opts.test.js
index.js

echo test/**/*.test.js
# zsh:
test/cli-opts.test.js
# bash 3.2:
test/**/*.test.js

echo {src,test}/**/*.test.js *.js
# zsh:
src/util/slurp.js 
src/cli-opts.js 
index.js 
test/cli-opts.test.js

# bash 3.2:
src/util/slurp.js 
index.js 
test/**/*.test.js
```

Now, if we look at `tape` and `eslint`'s respective `package.json` files, we'll see both use the [glob](https://github.com/isaacs/node-glob) package, which lets them accept glob patterns as operands, and to support the `**` globstar pattern in their expansion.

Bash's behavior on globs it was unable to match, that is to forward them unchanged as operands to the `tape` and `eslint` commands, is (counterintuitively!) the key thing that held the whole charade together, making bash work almost like zsh.

But not entirely. If you examine the expansions above closely, you may notice that in bash, we're not linting one set of files: anything directly under the `src/` folder. The linting command had quietly broke the moment I introduced the `src/util` subfolder, for reasons I'll leave as an exercise to the reader :-).

This subtle detail is why, for the most part, I hadn't realized anything was wrong with my setup; all tests were run and most files were linted. It was only when a recent mistake in `src/cli-opts.js` was caught by GitHub Actions, but not locally, that the jig was up. 

Why _did_ the `lint` command work in GitHub Actions? I have not had the energy to look into it, but my guess is the particular Ubuntu image I'm using may have bash version 4 or newer as its `/bin/sh`, with the `globstar` option enabled.

## The solution

The solution, as always, is to quote glob patterns to prevent their expansion in the shell, and have them delivered intact to the `tape` and `eslint` commands, which will in turn expand them consistently regardless of the particular shell they're running in.

__package.json__

```json
{
	"scripts": {
	  "test": "tape 'test/**/*.test.js'",
	  "lint": "eslint '{src,test}/**/*.js' '*.js'"
	}
}
```

I hope you found this write-up illuminating!