config: apply extends
Tags: javascript, config, showdev
Published: 2024-02-05 08:00:02

As we know, the forms and approaches to describing configuration files are extremely varied. I discussed this in [the previous note](https://dev.to/antongolub/topoconfig-enhancing-config-declarations-with-graphs-2d4). But there are several techniques and practices that may one day become standards.

One of this – the `extends` directive. Many tools provide this feature, but it works a little differently in each place. For example, `tsconfig` applies deep merge to `compilerOptions`, while `prettier` concatenates `overrides` array sections, [commitlint](https://github.com/conventional-changelog/commitlint/blob/79d77c9ef8de5046106027fc7d80dfd77faeb3e5/%40commitlint/resolve-extends/src/index.ts#L29) joins `plugin`,  etc. So the authors have to implement these nuances on site every time, and it's tiring and annoying a bit. I believe we will come up with a more efficient way to handle this routine.

## Proposal
```ts
import { populate, populateSync } from '@topoconfig/extends'

const tsconfig = await populate('tsconfig.json', {
  compilerOptions: 'merge'
})
const prettierCfg = populateSync('.prettierrc.json', {
  overrides: 'merge'
})
```

## 20 minutes adventure
At first glance, the `extends` may seem like a simple combination of `Object.assign` and `fs.readFile`. Well, technically it is, but not quite so. Let's see how the real scenario unfolds. To simplify the details, we will use pseudocode.

The starting point:
```ts
input = {
  foo: 'bar',
  extends: './base.json'
}
```

* pick `extends`
```ts
resource = input.extends
```

* load the referenced `./base.json`
```ts
extra = JSON.parse(fs.readFileSync(resource, 'utf8'))
```

* merge `input` and `extra`
```ts
result = {
  ...input,
  ...extra,
  extends: undefined,
}
```

How about `./base.js`? Reasonable request.
* provide js files support
```ts
input = { extends: './base.js' }
```

`require` hurries to the resque:
```ts
load = (ref) => isJs(ref) ? require(ref) : JSON.parse(fs.readFileSync(ref, 'utf8'))
```

`./base.mjs`?
* Ooops. Here the API switches to async mode to handle mjs/esm inputs.
  <sub><i>
  Of course, we can patch `require` with `esm` package, or load module via a `process.spawnSync`ed nodejs. We can, but we won't
  </i></sub>

```ts
load = isJs(ref)
  ? (await import(ref))?.default
  : fs.promises.readFile(ref, 'utf8')
```

What if the value is passed without an extension, like `base`?
* add support for shared npm configs
* provide configurable `resolve`
```ts
resolve = (ref, cwd) =>
  ref.startsWith('.') || ref.startsWith('/') || path.extname(ref)
    ? path.resolve(cwd, ref)
    : ref
```

But what if `extends` is a nested object? It definitely is.
* apply everything recursively:
```ts
handle = input = ({
  ...input,
  ...handle(load(input.extends)),
  extends: undefined,
})
```

In this case, circular references are something to pay attention to.
* add index to avoid infinite loops

```ts
memo = new Map()

memoizedLoad = ref => {
  if (memo.has(ref)) {
    return memo.get(ref)
  }
  memo.set(ref, load(ref))
  return load(ref)
}
```

Relative resource path should use the closest parent dirname as a base to work properly.
* resolve `cwd` for each `extends` reference
```ts
base = path.resolve(process.cwd(), _opts.cwd ?? '.')
[cwd, _config] = typeof config === 'string'
  ? [path.resolve(base, path.dirname(config)), path.basename(config)]
  : [base, config]
```

Multiple `extends` maybe?
* let `extends` hold several references

```ts
input = {
  foo: 'bar',
  extends: ['./extra.json', './mixin.js']
}
extras = [input.extends].flat().map(load)
```

How to read toml, yaml, ini, env? Obviously, custom formats are required too.
* provide `parse` option

```ts
parse = (ref, contents) => {
  if (ref.endsWith('.yaml') || ref.endsWith('.yml')) 
      return parseYaml(contents)
  if (ref.endsWith('.json')) 
      return JSON.parse(contents)
  throw new Error(`Unsupported format: ${ref}`)
}
```

How to resolve value conflicts: merge / override?
* provide `rules` to handle prop-specific join strategies
```ts
rules = {
  '*': 'merge',
  foo: 'override'
}
rule = getRule(propPath, rules)
result[key] = isObject(value) && rule === 'merge'
  ? extend({
    sources: [value],
    rules,
    prefix: p,
    index
  })
  : value
```

What about arrays?
* implement array merge as concat
```ts
if (rule === 'merge') {
  arr.push(...sources.flat(1))
} else {
  arr.length = 0
  arr.push(...sources.slice(-1).flat(1))
}
```

It would be nice to avoid mutations.
* introduce `clone`
```ts
safeLoad = compose(memoize, clone, load)
```

## Conclusion

Summarizing:
* search `extends` statements
* load & parse referenced resources
  * support async and sync flows
  * use cloning to avoid mutations
  * .js, .cjs, .mjs, .json
  * provide support for custom formats
  * resolve cwd for the nested resource by its closest parent resource
  * support custom resolvers
* walk recursively
  * handle circular references
* provide sync and async API
* assemble the result
  * handle arrays / plain objects
  * apply prop-specific merge

Additionally:
* build as cjs and esm
* provide types
* handle windows specific issues
* add dozens of tests
* handle different runtimes specifics (node, deno, bun)

There is an obvious benefit to implementing this algorithm on your own – it clarifies and makes controllable every aspect of config processing in place.
However, we do not always have enough time to do this exercise. Therefore, maybe someone will find this shortcut useful:

* https://github.com/antongolub/misc/tree/master/packages/topoconfig/extends
* https://www.npmjs.com/package/@topoconfig/extends