wj-config: Loading Configuration From Disk or Fetching From the Web
Tags: wjconfig, node, react, javascript
Published: 2022-10-24 02:08:47

> Originally published [here](https://webjose.hashnode.dev/wj-config-loading-configuration-from-disk-or-fetching-from-the-web).

> Heredia, Costa Rica
> 2022-10-23

> Series: One Configuration for all your JavaScript, Article 2

In this quick article I would like to present the various ways JSON data can be loaded from external sources, such as disks or from the web.

# Preface

**[wj-config](https://www.npmjs.com/package/wj-config)** is a JavaScript configuration package written in TypeScript and transpiled to *ES Modules* inspired by the .Net Configuration system where any number of data sources can be specified in the order of preference, and the end result is a merged version of all configuration data sources.

The supported data sources are:

1. Environment.
2. Raw JSON strings.
3. POJO's (Plain Old JavaScript Object).
4. Dictionaries (flat objects that follow a naming pattern).
5. Single values.
6. Fetched JSON data.

If so desired, anyone can add more data sources as the library is extensible.

The library provides many features.  Among those are:

1. An `environment` object that provides information about the current environment (development, staging, production, etc.) inspired by .Net's own `IHostEnvironment` interface.
2. Configuration value tracing (tells you which data source set the value found in the final configuration object).
3. URL functions that can automagically build URL's using the data found in the configuration object like host name, scheme, port and paths.  The functions even replace route values and dinamically support query string addition, all while URL-encoding all values.

**wj-config** can be used in browser frameworks like **ReactJS** as well as **NodeJS**.  One configuration library to rule them all.

# Import the JSON File

As seen in the first article of the series, one simple way to read a configuration JSON file is to simply import it.  This works in **NodeJS** as well as **ReactJS** and probably several other frameworks and libraries.  My current experience is based on the two aforementioned ones so this is what you'll see here.

## Importing in ReactJS

This is super simple, as shown in the first article of this series.

```javascript
import mainConfig from './config.json';
```

That's it.  Now your config file is parsed and its result stored in the variable named `mainConfig`.

## Importing in NodeJS

**NodeJS** has this additional syntax requirement:  An assert object that specifies the type of import.

```javascript

import mainConfig from './config.json' assert { type: 'json' };
```

The end result is the exact same thing:  `mainConfig` contains the contents of the JSON file.

This process, simple as it is, comes with drawbacks.  The most obvious one, and one that you probably can notice in the first article of the series, is that it is statically loaded.  You cannot opt-in, based on runtime code execution, whether or not you want or need the JSON data in said file.  This is particularly important when you want to conditionally load configuration data.  If you were to use this method to load environment-specific JSON's, you would have to load all your environment-specific JSON's, even though your system runs only under one of those environments.

# Load a JSON File From Disk

This method is applicable to **NodeJS** only because browser applications do not have access to the end user's file system, and even if they did, what's the point?  Your configuration file is not there.  Your configuration file probably resides in the server, so you better `fetch()` it (see next section).

Anyway, back on topic.  You can use **NodeJS**'s integrated `fs` module to load a JSON file.

This example demonstrates how to read and parse a JSON file, and then add it as configuration source:

```javascript
import fs from 'fs';
import wjConfig from 'wj-config';

function loadFile(fileName) {
    return new Promise((rslv, rjct) => {
        fs.readFile(fileName, 'utf8', (err, data) => {
            if (err) {
                rjct(err);
            }
            else {
                rslv(data);
            }
        });
    });
}

const fileContents = await loadFile('./package.json');
const contents = JSON.parse(fileContents);
const configPromise = wjConfig()
    .addObject(contents)
    .build();
const config = await configPromise;
console.log(config);
```

**NOTE**:  The example is intentionally written like this and not as an importable module merely so you can paste this code as the main code of a test **NodeJS** project and run it right away.  Usually we export `config`, then we import it where needed.

Now you can do things like an `if` statement to conditionally load a JSON configuration file and optionally include it, or build the JSON file's name in runtime.  You can even make the presence of said file optional and simply supply an empty object if the file is not found, like in the following variant:

```javascript
import fs from 'fs';
import wjConfig from 'wj-config';

function loadJsonFile(fileName, isRequired = false) {
    return new Promise((rslv, rjct) => {
        fs.readFile(fileName, 'utf8', (err, data) => {
            if (err && isRequired) {
                rjct(err);
            }
            else if (err) {
                rslv({}); // Empty object.
            }
            else {
                rslv(JSON.parse(data));
            }
        });
    });
}

const contents = await loadJsonFile('./package.json');
const configPromise = wjConfig()
    .addObject(contents)
    .build();
const config = await configPromise;
console.log(config);
console.log('End.');
```

**NOTE**:  This variant also includes an extra change:  Parsing is done inside the file-loading function, so the function is renamed to `loadJsonFile`.

Now if the file-loading code encounters an error (such as the file not existing), the returned promise still resolves successfully, only with an empty object.

This is my preferred method and how I recommend loading environment-specific data sources from disk (or fetched):  If not found, return empty objects so as to not disturb **wj-config**'s fluent syntax.

## About Dockerized NodeJS Applications

Dockerized containers usually don't have anything else running in them, so the file system is expected to be super stable and predictable.  This means that we can enhance the above function with `fs.existsSync()` to check if the file exists, and leaving the error for more serious matters only.

```javascript
function loadJsonFile(fileName, isRequired = false) {
    return new Promise((rslv, rjct) => {
        const noFile = !fs.existsSync(fileName);
        if (noFile && isRequired) {
            rjct(new Error(`File ${fileName} does not exist and is required.`));
        }
        else if (noFile) {
            rslv({});
        }
        else {
            fs.readFile(fileName, 'utf8', (err, data) => {
                if (err) {
                    // You probably want to log the error or something.
                    console.log(err);
                    if (isRequired) {
                        rjct(err);
                    }
                    else {
                        rslv({});
                    }
                }
                else {
                    rslv(JSON.parse(data));
                }
            });
        }
    });
}
```

# Fetch a JSON File

**NodeJS** has had an implementation of `fetch()` since some v17 release, and in v18, while still experimental is enabled by default and working fine in my limited testing.

Because of this, **wj-config** does come with built-in functionality to fetch JSON data and is very simple to use.

To add a data source object from fetched JSON data, you simply need something like this:

```javascript
import wjConfig from 'wj-config';

// In ReactJS, you could put configuration files in the root of the /public folder.
const myConfigUrl = '/config.json';

const configPromise = wjConfig()
    .addFetchedConfig(myConfigUrl)
    .build();
const config = await configPromise;
console.log(config);
```

Just like that.  I actually recommend that for **ReactJS** applications, you put the environment-specific configuration files in the `/public` folder, and maintain the main configuration file in the `/src` folder.  If you do this, you import the main configuration and you fetch the environment specific configuration.

```javascript
import wjConfig from 'wj-config';
import mainConfig from './config.json'

const myDevConfigUrl = '/config.dev.json';

const configPromise = wjConfig()
    .addFetchedConfig(myDevConfigUrl)
    .build();
const config = await configPromise;
console.log(config);
```

And that's about it for basic fetching, but let's explore a bit more some more advanced options available when fetching configuration data.

## The Overloads of addFetchedConfig

There are 2 overloads of the `addFetchedConfig()` method:

```typescript
addFetchedConfig(
    url: URL,
    required?: boolean,
    init?: RequestInit | undefined,
    procesFn?: ProcessFetchResponse | undefined
): IBuilder

addFetchedConfig(
    request: RequestInfo,
    required?: boolean,
    init?: RequestInit | undefined,
    procesFn?: ProcessFetchResponse | undefined
): IBuilder
```

Both overloads provide the same functionality and they differ in one thing only:  The data type of the `url` parameter (the first one).  They closely mimic `fetch()` because, after all, this is a wrapper of said function.

### Optionally Fetching

Probably the most helpful of parameters, besides the obvious `url` one, is the `required` parameter.  It is super simple to use:  Pass a Boolean value that indicates whether or not your application can only function properly if the data is fetched.  So basically pass `true` if you cannot live without this configuration data, or `false` if your application will be OK even if the fetching process fails.

If you passed `true` then **wj-config** will throw an error if the data cannot be fetched.  That simple.

### Passing Headers, Tokens and Whatnot

If your fetching of the configuration data requires special data like custom headers and such, you can make use of the `init` parameter.  Refer to the [MDN's documentation](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request) on `fetch()` to understand how it is used.  `addFetchedConfig()` does nothing with this except passing it to `fetch()`. **Hint**:  It is the `options` part in that MDN document.

### Custom-Process the Fetched Data

Ok, so most of the time you won't need this at all.  99.9% (or more) of the times, you'll be obtaining JSON data directly from the URL you specify in the call to `addFetchedConfig()` and that's it.  But what if the data comes in a different format?  What if it comes in JSON, but you only need pieces of that JSON?

Well, then you must provide your on data-treatment function.  This function of yours will receive the response object obtained by calling `fetch()` and it is up to you to do whatever you need to do to obtain the sought configuration data.

As an example, let's imagine that we want to fetch a large configuration document from a larger system, but we only need one or two sub-sections of that data.  No problem.  Do this:

```javascript
import wjConfig from 'wj-config';

const enterpriseConfigUrl = 'https://my-enterprise.api.example.com/config/full'; // Or whatever.

const configPromise = wjConfig()
    .addFetchedConfig(enterpriseConfigUrl, true, undefined, async (response) => {
        const data = await response.json(); // Obtain the full JSON response.
        // Now return only the needed pieces.
        return {
            subSection: data.clients.myClientsConfig
        };
        // Or spread.  Whatever works for you:
        return {
            ...data.clients.myClientsConfig
        }
    });
    .build();
const config = await configPromise;
console.log(config);
```

By providing the custom processing function we are able to download a big configuration data file and just pick the piece or pieces that we want.  In the example above we are taking just the configuration that we want for the specific client.

**HINT**:  This function can also be used to *transform* the hierarchy of the data into a hierarchy that is more of your liking.  After all, **wj-config** is about compiling configuration data in a hierarchical manner, and if you have a data source that does not conform to your preferred hierarchy, feel free to use this transformation function to create a version that does adhere to your likings/needs.

---

In the coming articles I'll cover the `Environment` class and how it can be used to stop hardcoding the environment-specific configuration file, like in the examples seen so far.  With that knowlegde you will be able to start using **wj-config** everywhere.