Learn JavaScript by building a UI framework: Part 4 - Creating A Module Bundler
Tags: bundler, node, javascript, npm
Published: 2019-09-19 17:29:20

---
title: Learn JavaScript by building a UI framework: Part 4 - Creating A Module Bundler
published: true
description: 
tags: bundler, node, javascript, npm
---

*This article is the fourth in a series of deep dives into JavaScript. You can view previous articles by visiting the [Github repository](https://github.com/CarlMungazi/aprender-js#what-is-this) associated with this project.*

*This series does not comprehensively cover every JavaScript feature. Instead, features are covered as they crop up in solutions to various problems. Also, every post is based on tutorials and open source libraries produced by other developers, so like you, I too am also learning new things with each article.*

___ 

In the [last article](https://dev.to/carlmungazi/learn-javascript-by-building-a-ui-framework-part-3-rendering-testing-dom-elements-97l) we added the functionality for our framework to create and render DOM elements, created an example application and then successfully tested it. Today we will cover the steps necessary to run our app in a browser. 

### The first step

At the moment, if we installed a server like [http-server](https://www.npmjs.com/package/http-server) and spun it up in the folder housing our example application, this error shows up in the browser console `Uncaught ReferenceError: require is not defined`. This is because the [`require`](https://nodejs.org/en/knowledge/getting-started/what-is-require/) function only exists in the node environment. It provides a way of accessing code that exists in separate files. The easiest (and most painful) way to replicate this behaviour in the browser would be to use `<script>` tags. 

Before the advent of [ES Modules](https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/), developers used (and still do) either the [CommonJS](http://eng.wealthfront.com/2015/06/16/an-introduction-to-commonjs/) or [AMD](https://www.devbridge.com/articles/understanding-amd-requirejs/) formats to tackle this problem. And this is where build tools such as [Webpack](https://webpack.js.org) or [Parcel](https://parceljs.org/) come in. Conceptually, their work is straightforward. They gather all the files needed to run an application, work out the dependencies of those files and then create one big JavaScript file which can run in a web browser. The complexity comes in the *how* of that process and various other cool tricks such as hot reloading (creating a new bundle each time you save changes to a file) and tree shaking (eliminating unused code). 


The first step in creating the bundler will be creating a command line tool so we can use [npm scripts](https://www.tutorialdocs.com/article/npm-scripts-tutorial.html) to initiate everything. Our framework `aprender` already has a `package.json` file so we begin by adding the following command.

```json
{
  "name": "aprender",
  "version": "1.0.0",
  "description": "",
  "main": "",
  "scripts": {
    "test": "node tests",
    "demo": "maleta demo/index.html --entry index.js"
  }
}
```
At this point it is worth exploring what happens when we type `npm run demo` in the terminal. Before running the commend, we first create a [`symlink`](https://medium.com/dailyjs/how-to-use-npm-link-7375b6219557) between `aprender` and our build tool, which will be called `maleta`. The symlink is created by:

- Creating a folder called `maleta` on the same folder level as `aprender`
- In the terminal, navigate to `maleta` and type  `npm link`
- Navigate to `aprender` and type `npm link maleta`

When `npm run demo` is executed, npm grabs the `scripts` object in aprender's `package.json` file and runs whatever command is assigned to the property `demo`. The first part of the demo command is referencing `maleta`, our module bundler. npm will [process](https://github.com/npm/cli/blob/v6.9.0/lib/run-script.js#L61) `maleta`'s `package.json` file and look for an object called `bin`. It looks like this: 

```js
"bin": {
  "maleta": "bin/cli.js"
}
```
The `bin` folder in any npm package contains executable files. The value of `maleta` is the path to the file `cli.js`, which contains the following code:


```js
#!/usr/bin/env node

const program = require('commander');
const version = require('../package.json').version;
const bundler = require('../src/bundler');

program.version(version);

program
  .command('serve <filename>')
  .description('serves the files')
  .option(
    '--entry <file>',
    'set the name of the entry JS file'
  )
  .action(bundle);

program
  .command('help [command]')
  .description('display help information for a command')
  .action(function(command) {
    let cmd = program.commands.find(c => c.name() === command) || program;
    cmd.help();
  });

const args = process.argv;

// Make serve the default command except for --help
if (args[2] === '--help' || args[2] === '-h') args[2] = 'help';
if (!args[2] || !program.commands.some(c => c.name() === args[2])) args.splice(2, 0, 'serve');

program.parse(process.argv);

function bundle (entryJsFile, command) {
  bundler(entryJsFile, {
    entryJsFile: command.entry
  });
}
```

This file is executed by your operating system's shell. npm does this by using node's [`child_process`](https://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options) method. The shebang `#!/usr/bin/env node` at the top of the file tells your operating system which interpreter or application to use when executing the file (if you are using Windows, this will be slightly different). When the node process is launched, any arguments specified are passed to the [`process.argv`](https://nodejs.org/docs/latest/api/process.html#process_process_argv) property. The first two arguments refer to the absolute pathname of the executable that started the process and the path to the JavaScript file being executed. Every argument from index two onwards is used by whatever code is being executed.

Maleta's CLI tool is built using [commander](https://github.com/tj/commander.js/). Commander exposes an object with a number of methods. We can use the `version` method to return the bundler version by typing  `maleta -V` or `maleta --version`. After that we use the `command` method to begin creating our commands. `command` takes one argument written in the following syntax `command <requiredArg> [optionalArg]`. Our CLI tool has two commands - one to serve the app and another to print help text. The string specified via `description` is displayed when a user runs the help command. The `action` method is used to specifiy the callback function which runs when the command is executed. It receives the argument(s) passed in via the `<>` or `[]` brackets and the commander object, which will have the names of any specified options among its properties. 

Taking inspiration from [Parcel](https://github.com/parcel-bundler/parcel/blob/v1.10.3/src/cli.js#L181), we make `serve` the default argument if no argument has been passed and then use commander's `parse` method to add the arguments to the commander object. Finally, `bundle` calls the imported `bundler` function with the entry file.


### The bundler at work

Maleta borrows much of its structure from [Minipack](https://github.com/ronami/minipack), a similar project written by [Ronen Amiel](https://github.com/ronami) that explains how bundlers work. The only differences are that Maleta bundles both ES and CommonJS modules, has a CLI tool and spins up a server to run the app. At the core of our bundler's work is the dependancy graph. This lists all the files used in an application along with any dependencies. Before building that graph, we will use the entry file to create a `rootAsset` object with the following structure:

```js
const rootAsset = {
  outDir: '', // the path of the directory where the bundle will created
  content: '', // the code in the file
  entryJsFilePath: '', // the path of the entry JavaScript file
  rootDir: '', // the path of the directory where the entry file lives
  dependencyGraph: '', // the dependencies of the entry file
  ast: '' // an abstract syntax tree created from the code in the file
}
```

Bundlers should be able to handle JavaScript or HTML files as the entry file but for simplicity, Maleta will only accept HTML files as the starting point. The function which creates the `rootAsset` object is:

```js
function createRootAssetFromEntryFile(file, config) {
  rootAsset.content = fs.readFileSync(file, 'utf-8');
  rootAsset.rootDir = getRootDir(file);
  rootAsset.outDir = path.resolve('dist');
  
  if (config.entryJsFile) {
    rootAsset.ast = htmlParser(rootAsset.content);
    rootAsset.entryJsFilePath = path.resolve(rootAsset.rootDir, config.entryJsFile);
  } else {
    extractEntryJSFilePathFromEntryFile(rootAsset);
  }

  rootAsset.dependencyGraph = createDependencyGraph(rootAsset.entryJsFilePath);

  return rootAsset;
}
```

It receives the arguments passed into the `bundler` function by the CLI tool. The only interesting activities  occur in the `htmlParser`, `extractEntryJSFilePathFromEntryFile` and `createDependencyGraph` functions. `fs` and `path` are node modules which are documented [here](https://nodejs.org/api/index.html) and `getRootDir` does what its name states. *Note: Reading the file synchronously with `fs.readFileSync` is not very performant as it a [blocking](https://nodejs.org/de/docs/guides/blocking-vs-non-blocking/) call but we are not too worried about that at this moment.*

When we call `htmlParser` it receives the following content from our demo app:

```html
<html>
  <head>
    <title>Hello, World</title>
  </head>
  <body>
    <div id="app"></div>
    <script src="./index.js"></script>
  </body>
</html>
```

`htmlParser` refers to the module [`posthtml-parser`](https://github.com/posthtml/posthtml-parser), a tool for parsing and turning HTML into an abstract syntax tree (AST). Our npm command `demo: maleta demo/index.html --entry index.js` helps us easily find the path to related entry JavaScript file. However, if the `--entry` option is missing, we call `extractEntryJSFilePathFromEntryFile`. 

```js
function extractEntryJSFilePathFromEntryFile(rootAsset) {
  const parsedHTML = htmlParser(rootAsset.content);
  
  rootAsset.ast = parsedHTML;
  parsedHTML.walk = walk;
 
  parsedHTML.walk(node => {
    if (node.tag === 'script') {
      if (node.attrs.src.endsWith('/index.js')) {
        rootAsset.entryJsFilePath = path.resolve(rootAsset.rootDir, node.attrs.src)
      }
    }

    return node;
  });

  if (!rootAsset.entryJsFilePath) throw Error('No JavaScript entry file has been provided or specified. Either specify an entry file or make sure the entry file is named \'index.js\'');
}
```
The only difference here is [`posthml`](https://github.com/posthtml/posthtml)'s `walk` method which we have attached to the AST. We use it to traverse the tree and ensure the HTML file has a link to a JavaScript file called `index.js`.

### Building the dependency graph

Our graph will be an array of objects listing every module in the application. Each object will have:

- an `id`
- the code from the module
- the original filename
- an array of the relative file paths of that module's dependencies 
- an object with the ids of those same dependencies. 

The first thing `createDependencyGraph` does is create the main asset from the entry JavaScript file using this function:

```js
function createJSAsset(filename) {
  const content = fs.readFileSync(filename, 'utf-8');
  const ast = babylon.parse(content, { sourceType: 'module' });

  const relativeFilePathsOfDependenciesArray = [];

  traverse(ast, {
    ImportDeclaration({ node }) {
      relativeFilePathsOfDependenciesArray.push(node.source.value)
    },
    CallExpression({ node })  {
      const { callee, arguments: args } = node;
      if (
        callee.name === 'require' &&
        args.length === 1 &&
        args[0].type === 'StringLiteral'

      ) {
        relativeFilePathsOfDependenciesArray.push(args[0].value)
      }
    }
  })

  const id = moduleID++;
  
  const { code } = transformFromAstSync(ast, null, {
    presets: ['@babel/env'],
    cwd: __dirname
  });

  return {
    id,
    code,
    filename,
    relativeFilePathsOfDependenciesArray,
    mapping: {}
  }
}
```

`babylon` is the same JavaScript parser used by [babel](https://babeljs.io/). Its `parse` method runs the given code as a JS program and in the second argument you pass an options object which tells it whether it is dealing with a module or script. Its output is an AST according to the [babel AST format](https://github.com/babel/babel/blob/master/packages/babel-parser/ast/spec.md). We use it with the babel plugin `traverse` ([babel-traverse](https://github.com/babel/babel/tree/master/packages/babel-traverse)) to find all the dependency references. [`ImportDeclaration`](https://github.com/babel/babel/blob/7f47cb65d0b7aea5f6c9a8ed0c255be1512738b7/packages/babel-parser/ast/spec.md#importdeclaration) finds all the ES module imports whilst [`CallExpression`](https://github.com/babel/babel/blob/7f47cb65d0b7aea5f6c9a8ed0c255be1512738b7/packages/babel-parser/ast/spec.md#callexpression) searches for every function call expression, from which we can check if its being done with the `require` keyword.

The next task is to parse the JavaScript code in the file. `transformFromAstSync` is a method from the [`babel/core`](https://babeljs.io/docs/en/next/babel-core.html#transformfromastsync) module and it turns our AST into the final code which will run in the browser. It also creates a source map. In the config object it is important to set the working directory to `maleta` otherwise any file paths will be resolved to whichever directory is running maleta, which in our case is `aprender`.  

Once the main asset has been created from the entry JavaScript file, it is assigned to the `assetQueue` array for processing. This array is a queue which will eventually contain assets representing every JavaScript file in the application. The relationship between each asset and its dependencies is stored in an object called `mapping`. Every property on this object is the file name of each dependency along with its id.

### Creating the bundle

```js
function createBundle(entryFile, config) {
  let modules = '';
  let bundle;
  const rootAsset = createRootAssetFromEntryFile(entryFile, config);
  const bundlePath = path.resolve(rootAsset.outDir, 'index.js');
  const bundleHtml = htmlRender(rootAsset.ast);
  const bundleHtmlPath = path.resolve(rootAsset.outDir, 'index.html');

  // ...
}
```

`createBundle` is the function used by our CLI to kickstart the bundling process. `createRootAssetFromEntryFile` performs all the steps listed above and returns a `rootAsset` object. From that, we create the file paths for the output files. We also use `htmlRender`(which is actually [`posthtml-render`](https://github.com/posthtml/posthtml-render)) to turn the AST we grabbed from the entry HTML file into a new HTML tree. The next step is to iterate over the dependency graph and create the bundled code like so:

```js
function createBundle(entryFile, config) {
  // ...

  rootAsset.dependencyGraph.forEach(mod => {
    modules += `${mod.id}: [
      function (require, module, exports) {
        ${mod.code}
      },
      ${JSON.stringify(mod.mapping)},
    ],`;
  });

  bundle = `
    (function(modules) {
      function require(id) {
        const [fn, mapping] = modules[id];

        function localRequire(name) {
          return require(mapping[name]);
        }

        const module = { exports: {} };

        fn(localRequire, module, module.exports);

        return module.exports;
      }

      require(0);
    })({${modules}})
  `;

  // ...
}
```

### The bundle explained

The bundle is an [immediately invoked function expression (IIFE)](https://developer.mozilla.org/en-US/docs/Glossary/IIFE), a JavaScript function that runs immediately as soon as it is defined. We assign it to the `bundle` variable and then pass in the `modules` object as the argument. Each module is an array with a function that executes code for that module as its first element and the module/dependency relationship as its second element.

The first thing the IIFE does is create a `require` function which takes an `id` as its only argument. In this function, we destructure the array and access the function and mapping object of each module. The modules will have `require()` calls to relative file paths and some might make calls to the same file paths even though they are referring to different dependencies. We handle that by creating a dedicated local `require` function which turns file paths into module ids. 

For example, in our demo application the `require(0)` call at the end of the IIFE results in the following:

```js
function require(id) {
  const [fn, mapping] = modules[id];
  /* the value of fn */
    function (require, module, exports) {
      "use strict";
      var aprender = require('../src/aprender');
      var button = aprender.createElement('button', {
        children: ['Click Me!']
      });
      var component = aprender.createElement('div', {
        attrs: {
          id: 'root-component'
        },
        children: ['Hello, world!', button]
      });
      var app = aprender.render(component);
      aprender.mount(app, document.getElementById('app'));
    }
  /* the value of mapping */ 
  {"../src/aprender": 1}
}
```
`require('../src/aprender');` is really `localRequire('../src/aprender')`. Internally, `localRequire` makes this recursive call `require(mapping['../src/aprender']`. `mapping['../src/aprender']` returns the value `1`, which is the `id` of the entry JavaScript file's only dependency. `require(1)` returns:

```js
function require(id) {
  const [fn, mapping] = modules[id];
  /* the value of fn */
    function (require, module, exports) {
      "use strict";
      var createElement = require('./createElement');
      var render = require('./render');
      var mount = require('./mount');
      module.exports = {
        createElement: createElement,
        render: render,
        mount: mount
      };
    }

  /* the value of mapping */
  {"./createElement":2,"./render":3,"./mount":4}
}
```

Each time the code in our dependencies makes a `require` call, it will be destructured in this way. The rest of the code in the bundler IIFE is: 

```js
function localRequire(name) {
  return require(mapping[name]);
}

const module = { exports: {} };

fn(localRequire, module, module.exports);

return module.exports;
```

`localRequire` wraps the recursive call we explained above and `fn(localRequire, module, module.exports)` executes the function we destructured at the beginning of the `require` function. All the exports from the dependencies of module in question will be stored in the `module` object. In our demo application, `createElement`, `render` and `mount` all export functions and an object with all these exports is value of the `aprender` module.

### Serving the bundle

Once the bundle is ready, we create an output directory, create the `index.js` and `index.html` files for the demo application and then serve them using [`http`](https://nodejs.org/api/http.html) and [`serve-static`](https://github.com/expressjs/serve-static).

```js
function createBundle(entryFile, config) {
  
  //...

  // create the output directory if it does not exist
  if (!fs.existsSync(rootAsset.outDir)) {
    fs.mkdirSync(rootAsset.outDir);
  }


  // create output html and js files
  fs.writeFileSync(bundlePath, bundle);
  fs.writeFileSync(bundleHtmlPath, bundleHtml);

  // create server and serve files
  const serve = serveStatic(rootAsset.outDir); 
  const server = http.createServer( function onRequest(req, res) {
    serve(req, res, finalhandler(req, res));
  });

  server.listen(3000);
  console.log(`${chalk.bold('Now serving the application on')} ${chalk.red('http://localhost:3000')}`);
}

```

### Summary

The bundler we created is by no means perfect and no doubt contains many holes and candidates for improvement. However, it is functional and that is the most important thing. We have reached a stage in our project where we can view our application in a browser. In the next article, we will return to our UI framework and add the functionality which allow us to create more complicated demo application.