Node.js inside-out - Modules API rediscovery...
Tags: node, javascript, api
Published: 2019-04-06 11:20:18

**This post is taken from [my blog](https://areknawo.com), so be sure to check it out for more up-to-date content 😉**

Since its first release in  **2011**, [**Node.js**](https://nodejs.org/) has greatly changed, or should I say  **revolutionized**JavaScript development and its use-cases. Being able to write code in their favorite language, and run in on the server-side, many web developer quickly notice the  **huge potential**  of the project. Fast forward to 2019 and Node.js is one of the most beloved and used  **runtimes**  in the entire programming market. It's fast (thanks to [**V8**](https://v8.dev/)), reliable and easy to use. And, with its own  **[NPM](https://npmjs.com/) package manager** 📦, it has the biggest collection of open source libraries and tools  **in the world**! These facts alone tell something about how popular Node.js has become. 🚀

For reasons above, in this series titled  **"Node.js inside-out"**! we're going to explore everything about the project. We'll talk about  **what Node.js**  actually  **is** and  **what APIs**  in-depth  **it provides**. With the rise of NPM and number of Node.js frameworks, many developers prefer that instead of the  **lower-level stuff**  that Node.js itself provides. 👏 Don't get me wrong - it's fine to use various tools that made your development more  **enjoyable**. It's just that sometimes, when you need to squeeze some  **additional performance**  or want to know what's going on under-the-hood, then it's good to get back to the basics. Sadly, many people omit this step when starting with Node.js (unlike the web development - JS, HTML and CSS are standard milestones for beginners), going straight to using different  **frameworks**  without really understanding them or their true potential.

With this intro finally behind us, as I said, I'd like this series to provide an alternative, more beginner-friendly  **Node.js super-in-depth introduction**  in slightly more... acceptable way. 😉 So, I hope you'll enjoy it and learn something new!

# What exactly is Node.js?

For start - a bit of theory... but not really boring one. 😃 Node.js itself is a  **runtime environment**for JavaScript. It's  **open-source**  and  **cross-platform**. Its development started in 2009, with the first official release in 2011. The idea behind it was simple - to allow JS to run in  **different environments**  than the browser. It's nothing that hasn't been done before, just not with that big success. Today its development is overseen by the  **Node.js foundation**  with additional help from a big number of contributors. It's used by many big names in the industry and doesn't seem to stop evolving and improving with time.

As a runtime environment, Node.js is powered by  **V8** 🔋 - open-source JS engine (the fastest on the market), originally created  **by Google**. Thus, it's working similarly to any  **Chromium-based**browser. The code is run in an  **event loop**, on a  **single thread**. The  **asynchronous I/O**  allows it to take care of multiple  **concurrent operations**. ⚡ This approach has its downsides, but they're related to JS in general.

Node.js also provides a lot of  **APIs** for developers to use. They allow accessing features that aren't possible through everyday browsers. They're provided in the form of  **modules**, as a standard way to handle core functionalities. Their features vary greatly - from  **file system**  access and  **cryptography**  to  **C++ add-ons**,  **child processes**, and  **V8 access**. We'll explore each of these later on in the series. 👍

With Node.js rapid development, more and more interesting tools appeared. With its robust architecture, you can create  **server-side code**,  **CLI tools**,  **real-time applications**, which further means likes of games, social media and others! Of course, it's all possible with the combination of client-side, which can be written, utilizing your current knowledge, in JS too! But I don't only mean browsers! Based on, inspired by or built with Node.js, tools like  [**Electron**](http://electron.atom.io/) or [**NativeScript**](https://www.nativescript.org/) came to exist. Basically what they do is allow you to create fully native desktop or mobile applications... with JavaScript! And, IMHO, that was the key to Node.js success - one,  **single language to rule 'em all**! ✊

# Node.js APIs

I'd like to commit the rest of this series to explore different APIs of Node.js. We'll start with the  ([**ECMAScript**](https://nodejs.org/docs/latest-v10.x/api/esm.html))  [**Modules API**](https://nodejs.org/docs/latest-v10.x/api/modules.html). I think that's a good-enough choice for the beginning. 😃 Modules are used almost everywhere, but you could be surprised by how many (possibly)  **unknown features**  they possess in Node.js. But, if you think this is too easy, then fear not! We'll explore more advanced APIs in future posts! And, as a side-note - I'm using  **Node.js v10.15.3**  - latest stable  [**LTS version**](https://nodejs.org/en/about/releases/)  for the rest of this tutorial. Now, let's get started! 🎉

# Modules

Probably almost all of today's web developers use some kind of module system to better organize their code. The most popular options being ES modules (newest standard) and CommonJS format (the one used in Node.js). But there's a little more to Node.js module system that just importing and exporting stuff. 😅 And that's what we're going to explore!

## CommonJS

Let's first recall the rules of  **CommonJS**  (**CJS**) module format - the main one in Node.js. In Node.js, unlike in any other front-end TypeScript or Babel-based workflow, modules are real things. Your imports and exports are  **resolved at runtime**  - not at any kind of transpilation step. Your basically getting is a  **real module system**. 😮 This, naturally, has its pros as well as cons. But, transpliation is still a nice option to have (especially when, e.g. doing micro-optimizations and not wanting to resolve modules at runtime) you can easily use Babel or any other tool you want - anytime anyhow! 😉

I guess many people refer to CJS as the one with  `require()`  **syntax**. That's because this particular keyword is probably the most recognizable symbol for this format.

### Import / export

For exporting you can assign your value to the special  `module.exports`  property to respective properties when dealing with objects. For the second purpose, you can also use the  `exports`  object - a quick shortcut. Just don't mess the two when assigning single values - exports won't work with stuff like that! That's because  `exports`  is, in fact, a reference to  `modules.exports`, which defaults to  **empty object**.

```javascript
// module1.js
exports.numberValue = 10;
exports.stringValue = 'str';
```

```javascript
// module2.js
module.exports = () => {
   // code 
}
```

Notice, that the  **arrow function**  ➡ syntax (and many other  **ES-Next features**) is  **natively supported**  by Node.js (and other V8-based browsers).

Imports can be done with well-known  `require()`  syntax:

```javascript
const module1 = require('module1.js');
const module2 = require('module2.js');

module1.numberValue; // 10
module1.stringValue; // 'str'
module2();
```

I think it's a well-known fact that the syntax above can be freely used to import core Node.js modules (like  **fs**  or  **path**), modules located in relative paths (  `./`  ),  **node_modules**  directory, but also the  **global**  ones. Also, you can feel free to omit your  **.js**,  **.json**  or  **.node**  (for native add-ons) file extensions. Or use the  **index.js**  files as folders' main files and etc. Just the usual stuff related to JS modules. Most of the times, it goes unnoticed... ⚡

### Wrappers & globals

Everything above is just pure basics. You can easily go and use that without any further understanding. But, in this series, we're digging deep! And so, we want to know what  `require()`,  `module`  and  `exports`  really are.

Before execution, code from each imported module is put inside a  **wrapper function** 🌯, looking something like this:

```javascript
((exports, require, module, __filename, __dirname) => {
    // module code
});
```

This is a very important concept to understand, and that's for 2 main reasons:

1.  All what-seems-like global variables and other user-defined variables at the top scope of different modules are preserved in limited,  **module-only scope**. You have to use  `module.exports`/  `exports`  to actually  **output something**  to the outer world. 📤
2.  This perfectly shows us where our  `require()`  function and  `module`  object actually come from. It also hides from developers the fact of function wrappers in a nice form of  **what-seems-like globals**. 👍

With that said, I think it's a perfect time to explore what parameters of our top wrapper really do:

-   **exports**  - just a reference to  `module.exports`  (as said before);
-   **require()**  - function used to  **import modules**. It has some additional properties of its own:
    -   **cache**  - object where all loaded modules are  **cached**  (more on that later);
    -   **main**  - reference to a  `Module`  **object**  representing  **entry module**;
    -   **resolve()**  - returns the  **exact filename**  (complete path) to the file that the module would be imported from when using  `require()`  with the same argument:
        -   **paths()**  - returns an array of  **paths searched through**  when locating the provided module;
-   **module**  - a reference to the object (Module instance) representing the  **current module**:
    -   **children**  - an array of modules first  **imported**  in the given module;
    -   **exports**  - an object used to  **export**  values from the given module;
    -   **filename**  -  **absolute path**  to the given module;
    -   **id**  -  **identifier**  for the given module. Usually equal to the filename (example exception being index files);
    -   **loaded**  - indicating whether the  **module has already loaded**. Especially important when using multiple requires in different places i.e. cycles. Ensure that they're properly loaded first;
    -   **parent**  - reference to the module that has loaded given module first;
    -   **paths**  - an array of paths searched through when locating the given module;
    -   **require()**  - provides a way to call require as if it was from the given module;
-   **filename**  - an  **absolute path**  of the module;
-   **dirname**  -  **directory name**  of the module;

Feels a bit like docs, does it? 😅 Hopefully it's not bad. I tried to provide it in a form that's both more understandable,  **shorter**  and simpler than the official documentation. The main point is just to understand where these seeming globals come from and what do they do. You'd most likely hardly-ever use any of properties above. Noticeable exceptions (beyond import/export syntax) include`__dirname`  and  `__filename`  which many beginners may not know where they come from and what do they represent. Well, now you know. 😉

**The Module**  (capped letter on purpose) is a structure that all modules instances mentioned above inherit from. Node.js allows you to access this as well, in a form of core  **module**  module 😂 (`require('module')`). It has even fewer use-cases than the API above, as it provides only two additional properties:

-   **builtinModules**  - an array of Node.js built-in modules' names;
-   **createRequireFromPath()**  - allows creating a  **relative requires**  that resolves to start from provided path, e.g. folder. Useful when using multiple imports from the same directory while not limiting readability;

As you can see, the properties above have their really, really specific use-cases. As such, I'd consider them more as internal properties rather than general-use ones. 😅 Although, if you're developing a Node.js framework... who knows? 🤔

### Caching

The last thing to note about modules is that they're  **cached**. This has a huge impact on how they work and the  **performance**  of actually loading them. Once loaded, your module won't have to be reloaded the second time. Instead, its cached version will be used (stored in object referenced by  `require.cache`). This results in improved performance, but also has some additional, sometimes taken-as-granted,  **side-effects**. You see, when a module is first loaded (that's why the children and parent properties of module exist BTW, because they indicate specific relations between modules, i.e. where it was first loaded and thus cached), cached and then accessed, all of its code has been  **executed once**  and all exports of this module are  **carried throughout all files**  that imported given module. This allows for some cunning tricks, like a dedicated module for semi-globals (values that can be imported anywhere and changed, affecting other modules). 🛸

Of course, you can force reload of a module by messing with  `require.cache`  object and removing given module (by its id). But, it's  **not really recommended**  practice - unless you're sure that this is exactly what you want.

## ECMAScript Modules

Up to this point, we were talking only about CJS modules. But, as many web developer should now, there has been a new standard introduced in 2015 with  **ES6**  (not that new any longer, huh? 😅) which is referred to as  **ECMAScript Modules**  (**ESM**  for short). They're the ones who brought us this fine  `import`/`export`  syntax and finally an  **industry-grade standard**! Sadly, as we've already seen with Node.js itself, the old standards (CJS, AMD, etc.) still prevail in some, even as actively-developed places as Node. But, this has finally changed with the release of  **Node v8.x**where support for ESM was introduced, although with an  **experimental flag**  ☢ (which stayed to current v11.x so far). But, that's not something that would stop us from taking a closer look at ESM in Node.js, is it? 👍

### Enable

As much as experimental status may not bother you (apart from some features still needed to be implemented or improved), it comes with some additional requirements. ESM (at the time of writing) isn't supported out-of-the-box. You have to use  `--experimental-modules`  flag to properly enable them whenever running Node. Also, you have to use the  **.mjs**  extension for your files to be properly loaded through ESM system. 👉

Quite frankly, ESM is mostly  **backward-compatible**  with CJS (with some API differences), meaning that you can freely import CJS modules through ESM without much hassle. On the other hand, what you cannot do, is importing ESM modules with CJS syntax. This is not allowed, as CJS uses  **different resolving method**  and  **timing**  (not forward-compatible 😅). Of course, the  **JSON files**  and C++ modules/**native addons**  can freely be used with the ESM syntax.

### Differences

Beyond cross-compatibility, there are a couple more differences between Node.js CJS and ESM implementations. ESM has completely different  **resolving system**, based on  **URL**  and  **file: protocol**. This means that you can e.g. pass additional  **query parameters**  to indicate that the following module should be loaded again (instead of using its cached version). 💾

```javascript
import module from './module1.js?id=1';
import moduleClone from './module1.js?id=2';
```

For now, the external URL cannot be used. Although with schematics above, it may be possible in the near future.

The URL format is also used to identify modules inside cache (that's why the example above works). But, as we don't have access to the same values available to us as in CJS (`require()`,  `module`, etc.) the cache object is stored separately. Also, unlike CJS, ESM  **doesn't resolve NODE_PATH**, which further means no way of importing globally-installed modules.

And finally, at its current state,  `import`  provides one property of its own. It's an object called  `import.meta`  which, again, has one property called  `import.meta.url`, indicating the  **absolute URL**  of the current module.

```javascript
import.meta.url
```

### Hooks

The last new feature of Node.js ESM is called  **loader hooks**. ⚡ As the name suggests, these hooks allow you to  **intercept the loading process**  of ESM modules with your own,  **custom code**. 👏

There are 2 possible hooks for you to use -  `resolve()`  and  `dynamicInstantiate()`. You can provide one or both of those in a form of  **asynchronous functions**, in a single, separate JS file. You later can load and use them with a simple CLI argument:

```bash
node --experimental-modules --loader ./loader.mjs ./index.mjs
```

The  `resolve()`  hook takes 3 parameters:

-   **specifier**  - an  **absolute path**  of the current module's file;
-   **parentModuleURL**  -  **URL of the parent module**  (the one that loaded given module first). It follows file: protocol and defaults to undefined when used on the entry module (there's no parent);
-   **defaultResolve()**  - default  **resolve function**;

After appropriate processing, your resolve hook should  **return an object**  with two properties:  `url`and  `format`. The first indicates the URL resolved for the handled module (**file:**) and second -  **module's format**. 📦 While  `url`  is a no-brainer,  `format`  has a form of a string with 6 possible values:

-   **"esm"**  - indicates ESM module;
-   **"cjs"**  - indicates CJS module;
-   **"builtin"**  - indicates Node.js built-in modules, e.g. http or path;
-   **"json"**  - indicates JSON file;
-   **"addon"**  - indicates a C++ native addon;
-   **"dynamic"**  - indicates the use of dynamicInstantiate hook;

The  `dynamicInstantiate()`  hook allows you to properly handle modules with  `"dynamic"`  format. The hook itself is an  **async function**  taking a single  `url`  argument (URL of the handled module), that should return an object with 2 properties:

-   **exports**  - an array of names for exported properties;
-   **execute()**  - functions taking above exports as an argument. It should access the previously defined property names on exports object and interact with them using .get() and .set() methods accordingly. It will be later executed at the time of  **module evaluation**;

In general, this hook gives you an option to provide a somewhat alternative form for modules that require that (e.g. different file extensions). Just keep in mind that it doesn't have to be limited to just setting completely different properties - you can use the provided URL to load and evaluate the file the way you want. As always in programming -  **options are almost* endless**! 😉

# We're just getting started!

Yup, it's been a while and we only managed to cover  **Modules API**  - just bare modules! Seemingly such a simple thing and has so much depth to it! 🤔 Again, don't worry, there's some even more  **interesting stuff in stock**! I'm planning on covering the  **File System API**  next (that's the big one!), but maybe you'd like to see something else? I'm very much open to different options! And remember that I plan on covering all Node.js APIs eventually!

So, let me know  **down in the comments**  what do you think about this article and what would you like to see next! Also,  **share this post**  with other for reach! 😃 As always,  **follow me**  [**on Twitter**](https://twitter.com/areknawo),  [**on my Facebook page**](https://www.facebook.com/areknawoblog)  and consider checking out [**my personal blog**](https://areknawo.com) to keep up-to-date with  **the latest content**  about this series and other awesome JS stuff! 🛸 Thanks for reading and I see you  **in the next post**! ✌