Discovery.js tutorials: quick start
Tags: discoveryjs, tutorial, node, javascript
Published: 2019-09-16 12:35:47

This and the following tutorials will guide you through the process of building a solution based on [discoveryjs](https://github.com/discoveryjs) projects. As a goal we will get a NPM dependencies inspector, i.e. an interface for exploring structure of `node_modules`.

> Note: Discoveryjs is at an early stage, so some things might to be changed to be more useful and simpler. If you have an idea how to make something better, let us know on [discoveryjs issues](https://github.com/discoveryjs/discovery/issues).

## TL;DR

Below you will find an overview of `discoveryjs` key concepts. You can explore all the sources from the tutorial combined in [a repo on GitHub](https://github.com/discoveryjs/quick-start-tutorial) or [try how it works online](https://discoveryjs.github.io/quick-start-tutorial/).

## Prerequisites

Before we start, we need a project to analyze. It may be a new project or an existing one, the only requirement is that it has a `node_modules` inside (a subject of our analysis).

As a first step, we need to install `discoveryjs` view and CLI tools:

```bash
npm install @discoveryjs/discovery @discoveryjs/cli
```

As our next step, we need to launch a discovery's server:

```
> npx discovery
No config is used
  Models are not defined (model free mode is enabled)
Init common routes ... OK
Server listen on http://localhost:8123
```

And open `http://localhost:8123` in a browser to see something:

![Discovery with no setup](https://thepracticaldev.s3.amazonaws.com/i/fg2nuy238e8xm9orls7z.png)

That's a model-free mode, where nothing is pre-configured. You can choose any JSON file via "Load data" button or drop it right on the page, and start exploring it.

However, we need something specific, in particular, we need to get a `node_modules` structure. Let's add some configuration.

## Add a configuration

As you might have noticed, there was a message `No config is used` when we first launched the server. So let's create a config file named `.discoveryrc.js` with following content:

```js
module.exports = {
    name: 'Node modules structure',
    data() {
        return { hello: 'world' };
    }
};
```

Note: If you're creating a config file in current working directory (i.e. in a root of project), then no additional action is needed. Otherwise, you need to pass a path to the config file with `--config` option, or specify it in `package.json` this way:

```json
{
   ...
   "discovery": "path/to/discovery/config.js",
   ...
}
```

Ok, let's restart the server to apply a config:

```
> npx discovery
Load config from .discoveryrc.js
Init single model
  default
    Define default routes ... OK
    Cache: DISABLED
Init common routes ... OK
Server listen on http://localhost:8123
```

As you can see, a config file that we created is being used now. And there is a default model, which we defined (discovery can run in multi-model mode, we'll cover this approach in later tutorials). Let's see what we get in the browser:

![With basic configuration](https://thepracticaldev.s3.amazonaws.com/i/0p2p7yuy05qd99vmwoi2.png)

What's we see here:
- `name` is used as a header of the page;
- result of `data` method invocation is displayed as main content of the page

> Note: `data` method must either return data or a promise, that will be resolved to data.

Our basic setup is ready, now can move on to the next step. 

## Context

Before moving forward, let's look on report page (click `Make report` to open it):

![A report page](https://thepracticaldev.s3.amazonaws.com/i/9c2glc0bvztmbimv25j9.png)

At first glance, that's the same as index page... But we can change everything! For example, we can recreate an index page, that's easy:

![Recreating an index page](https://thepracticaldev.s3.amazonaws.com/i/ro3o3weiq4lwo5jy0kf8.png)

Notice how a header is defined: `"h1:#.name"`. That's a header level 1 with `#.name` as a content, which is a [Jora](https://github.com/discoveryjs/jora) query. `#` is referencing to a context of query. To see what it contains just enter `#` in a query editor and use default view:

![Values of context](https://thepracticaldev.s3.amazonaws.com/i/kc35dqs5qw05n3mmczkr.png)

So now you know where you can get a current page ID, its params and other stuff.

## Collecting data

Back to our project, currently we used a mock data, but we need to collect real data. So we should create a module and change `data` value in the config (btw, you should not to restart a server after such changes):

```js
module.exports = {
    name: 'Node modules structure',
    data: require('./collect-node-modules-data')
};
```

The source of `collect-node-modules-data.js`:

```js
const path = require('path');
const scanFs = require('@discoveryjs/scan-fs');

module.exports = function() {
    const packages = [];

    return scanFs({
        include: ['node_modules'],
        rules: [{
            test: /\/package.json$/,
            extract: (file, content) => {
                const pkg = JSON.parse(content);

                if (pkg.name && pkg.version) {
                    packages.push({
                        name: pkg.name,
                        version: pkg.version,
                        path: path.dirname(file.filename),
                        dependencies: pkg.dependencies
                    });
                }
            }
        }]
    }).then(() => packages);
};
```

I used `@discoveryjs/scan-fs` package, which simplify file system scanning by defining rules. In package's readme an usage example can be found, so I took it as a basis and reworked as needed. And now we have some info about `node_modules` content:

![Data collected](https://thepracticaldev.s3.amazonaws.com/i/zyxa4ebpqwe8qkd3v7ov.png)

Much better! Despite it's just a JSON, we can dig into it and gain some insights. For example using signature popup we can found out a count of packages and how many of them has more than one physical instance (due to different versions or problems with package deduplication).

![Exploring data structure](https://thepracticaldev.s3.amazonaws.com/i/j78eccneotspsgdyvb44.png)

Although we have got some data, we need more details. For example, it's good to know to which physical instance resolves every of dependencies. Nevertheless improvements on data fetching is out of scope of this tutorial. So we just take `@discoveryjs/node-modules` (which also built on `@discoveryjs/scan-fs`) and get most details about packages with ease. `collect-node-modules-data.js` simplifies dramatically:

```js
const fetchNodeModules = require('@discoveryjs/node-modules');

module.exports = function() {
    return fetchNodeModules();
};
```

And data about `node_modules` is now looks like this:

![New data structure](https://thepracticaldev.s3.amazonaws.com/i/9376bzwytjnkjeyn1wgo.png)

## Preparation script

As you may have noticed, some packages contains `deps` – a list of dependencies. Each dependency has `resolved` field, where value is a reference to a package physical instance. The reference is a `path` value of one of the packages, since each `path` value is unique. To resolve a reference to a package, we need to use an additional code (e.g. `#.data.pick(<path=resolved>)`). But for sure, it would be much suitable to have such references already resolved.

Unfortunately, we can't resolve references on data collecting stage, as this will lead to circular references and data duplication, as well as making data transfer problematic. Nevertheless, there is a solution for this – a special script called `prepare`. That script is defined in config, and is invoked for any new data for discovery instance. Let's start with config:

```js
module.exports = {
    ...
    prepare: __dirname + '/prepare.js', // Note: value is a path to a module
    ...
};
```

And then define a `prepare.js`:

```js
discovery.setPrepare(function(data) {
    // do something with data or/and with discovery instance
});
```

In this module we specified `prepare` function for a discovery instance. This function invoked every time before data is applied to discovery instance. That's a good place to resolve references:

```js
discovery.setPrepare(function(data) {
    const packageIndex = data.reduce((map, pkg) => map.set(pkg.path, pkg), new Map());

    data.forEach(pkg =>
        pkg.deps.forEach(dep =>
            dep.resolved = packageIndex.get(dep.resolved)
        )
    );
});
```

Here we create a package index, where key is package's `path` value (which is unique). After that we go through all packages and each dependency, and replace `resolved` value for a reference to a package. That's a result:

![Resolved deps.resolved](https://thepracticaldev.s3.amazonaws.com/i/emc4ye3xlwzmbzjfuicj.png)

It's much easier to make queries to dependency graph now. Here's how to get a dependency cluster (dependencies, dependency dependencies, etc.) for a specific package:

![Example of dependency cluster](https://thepracticaldev.s3.amazonaws.com/i/5oy1713lfd2wg0ygex0k.png)

> Unexpected success story: exploring tutorial's data I found a problem in`@discoveryjs/cli` (with the query `.[deps.[not resolved]]`), it had a typo in a peer dependency reference. This has been [fixed](https://github.com/discoveryjs/discovery-cli/commit/3f300cedb7e8217d339cb0ecd7aa57e3e53ecd91) immediately. This is a case in point.

I suppose, it's a good time to show up some numbers and packages with duplicates on index page.

## Setup default page

First of all we need to create a page module, e.g. `pages/default.js`. `default` is used since index page has that slug and we can override it (most things in discoveryjs can be overridden). We might start with something simple, like this:

```js
discovery.page.define('default', [
    'h1:#.name',
    'text:"Hello world!"'
]);
```

Now we need to link the module in the config:

```js
module.exports = {
    name: 'Node modules structure',
    data: require('./collect-node-modules-data'),
    view: {
        assets: [
            'pages/default.js'  // a reference to page's module
        ]
    }
};
```

Checking up in a browser:

![Overridden default page](https://thepracticaldev.s3.amazonaws.com/i/k8feoq2gwo4yho24xlb9.png)

It works!

Let's show some counters, by changing `pages/default.js` this way:

```js
discovery.page.define('default', [
    'h1:#.name',
    {
        view: 'inline-list',
        item: 'indicator',
        data: `[
            { label: 'Package entries', value: size() },
            { label: 'Unique packages', value: name.size() },
            { label: 'Dup packages', value: group(<name>).[value.size() > 1].size() }
        ]`
    }
]);
```

Here we define an inline list of indicators. A `data` value is a Jora query, that produces an array of entries. A package list is used as a data source (a data root), so we get a list length (`size()`), a number of unique names (`name.size()`) and number of groups by a name that has more than a single member (`group(<name>).[value.size() > 1].size()`).

![Add indicators on index page](https://thepracticaldev.s3.amazonaws.com/i/cf4w97i3kkv9b8matg02.png)

Not bad. However, it would be better to have a link to selected entries besides numbers:

```js
discovery.page.define('default', [
    'h1:#.name',
    {
        view: 'inline-list',
        data: [
            { label: 'Package entries', value: '' },
            { label: 'Unique packages', value: 'name' },
            { label: 'Dup packages', value: 'group(<name>).[value.size() > 1]' }
        ],
        item: `indicator:{
            label,
            value: value.query(#.data, #).size(),
            href: pageLink('report', { query: value, title: label })
        }`
    }
]);
```

First of all `data` value was changed, now it's a regular array with a few objects. In addition `size()` method was removed for each value queries.

Also subquery was added to `indicator` view. Such queries produce a new object, where `value` and `href` property values are computing. For `value` it performs a query using `query()` method and pass data to it from a context, then apply `size()` method to a query result. For `href` it uses `pageLink()` method to generate a link to report page with specific query and title. After those changes indicators became clickable (notice that their values became blue) and much functional.

![Clickable indicators](https://thepracticaldev.s3.amazonaws.com/i/smfew510zs8di508kkej.gif)

To make index page a bit useful, let's add a table with duplicated packages.

```js
discovery.page.define('default', [
    // ... the same as before

    'h2:"Packages with more than one physical instance"',
    {
        view: 'table',
        data: `
            group(<name>)
            .[value.size() > 1]
            .sort(<value.size()>)
            .reverse()
        `,
        cols: [
            { header: 'Name', content: 'text:key' },
            { header: 'Version & Location', content: {
                view: 'list',
                data: 'value.sort(<version>)',
                item: [
                    'badge:version',
                    'text:path'
                ]
            } }
        ]
    }
]);
```

The same data as for `Dup packages` indicator is used for the table. Additionally, a package list was sorted by a group size in reverse order. The rest setup is for columns (btw, often you don't need to setup them). For `Version & Location` column we defined a nested list (sorted by a version), where each item is a pair of version badge and path to instance.

![Added a table of packages with more than one physical instance](https://thepracticaldev.s3.amazonaws.com/i/2g74fgg6svq8iyahprs4.png)

## A package page

Currently we have only overall view of the packages. It might be useful to have a specific package page. To archive this we need to create a new module `pages/package.js` and define a new page:

```js
discovery.page.define('package', {
    view: 'context',
    data: `{
        name: #.id,
        instances: .[name = #.id]
    }`,
    content: [
        'h1:name',
        'table:instances'
    ]
});
```

In this module we define a page with slug `package`. As a root view `context` view is used, that's a non-visual view which helps define common data for nested views. Notice that we use `#.id` to get a package name, that comes from an URL, i.e. `http://localhost:8123/#package:{id}`.

Don't forget to include new module to the config:

```js
module.exports = {
    ...
    view: {
        assets: [
            'pages/default.js',
            'pages/package.js'  // here you go
        ]
    }
};
```

And here is a result in a browser:

![A package page example](https://thepracticaldev.s3.amazonaws.com/i/57ph7siuanhx9kqsoygz.png)

It's not so impressive, but OK for now. More complex views will be created in next tutorials.

## Sidebar

Since now we have a package page, it's nice to have a list of all the packages. We might to define a special view `sidebar` for this, which renders when defined (is not defined by default). Let's create a new module `views/sidebar.js`:

```js
discovery.view.define('sidebar', {
    view: 'list',
    data: 'name.sort()',
    item: 'link:{ text: $, href: pageLink("package") }'
});
```

Now we have a sidebar with all the packages:

![Added sidebar](https://thepracticaldev.s3.amazonaws.com/i/d2zyu9uxr7cwn38g8zyg.png)

Looks good. But with a filter it might be much user friendly. Extending `sidebar` definition:

```js
discovery.view.define('sidebar', {
    view: 'content-filter',
    content: {
        view: 'list',
        data: 'name.[no #.filter or $~=#.filter].sort()',
        item: {
            view: 'link',
            data: '{ text: $, href: pageLink("package"), match: #.filter }',
            content: 'text-match'
        }
    }
});
```

Here we're wrapped the list into `content-filter` view, which will provide an input value converted to RegExp (or `null` when empty) as `filter` value in the context (a name may be changed via `name` option). Also we used `#.filter` to filter data for the list. Finally, a link view definition was extended to highlight matching parts using `text-match` view. And there is a result:

![Sidebar with a filter](https://thepracticaldev.s3.amazonaws.com/i/t98xdlq0v8r5s8pauihy.png)

In case you dislike default style of something, you may tweak styles as you want. Suppose, you want to change a sidebar width. Then you need to create a style file (`views/sidebar.css` would be a good choice):

```css
.discovery-sidebar {
    width: 300px;
}
```

And to include a reference to this file in the config, just like with JavaScript modules:

```js
module.exports = {
    ...
    view: {
        assets: [
            ...
            'views/sidebar.css',  // you may specify *.css files in assets too
            'views/sidebar.js'
        ]
    }
};
```

## Auto linking

Last chapter of this tutorial is about links. As you can see above, we made a link to a package page via `pageLink()` method. Beside that we need to specify link text as well. But how about make it a bit simpler?

To simplify linking, we need to define a link resolver. A good place for this is `prepare` script:

```js
discovery.setPrepare(function(data) {
    ...

    const packageIndex = data.reduce(
        (map, item) => map
            .set(item, item)        // key is item itself
            .set(item.name, item),  // and `name` value
        new Map()
    );
    discovery.addEntityResolver(value => {
        value = packageIndex.get(value) || packageIndex.get(value.name);

        if (value) {
            return {
                type: 'package',
                id: value.name,
                name: value.name
            };
        }
    });
});
```

We added a new map (an index) for packages here, and used it for an entity resolver. The entity resolver makes an attempt to translate passed value into a package descriptor when possible. A package descriptor contains:
- `type` – slug of instance type
- `id` - unique reference to an instance, used as page ID in links
- `name` – used as captions for links

Last step, we need to attach this type to certain page (a link should lead somewhere, isn't it?).

```js
discovery.page.define('package', {
    ...
}, {
    resolveLink: 'package'  // link `package` entities to this page
});
```

The first effect of those changes, is that some values in `struct` view now marked with a badge link to the package page:

![Auto-link in struct](https://thepracticaldev.s3.amazonaws.com/i/pajh5110lvykh50iax4v.png)

And now you may apply `auto-link` view for an package object or name:

![Using auto-link](https://thepracticaldev.s3.amazonaws.com/i/6zw8apz0zms4kdg3wson.png)

As an example, sidebar can be slightly refactored:

```js
    // before
        item: {
            view: 'link',
            data: '{ text: $, href: pageLink("package"), match: #.filter }',
            content: 'text-match'
        },

    // with `auto-link`
        item: {
            view: 'auto-link',
            content: 'text-match:{ text, match: #.filter }'
        }
```

## Conclusion

Now you have a basic knowledge of `discoveryjs` key concepts. Next tutorials will continue to guide you through these topics in more depth.

You can explore all the sources from the tutorial combined in [a repo on GitHub](https://github.com/discoveryjs/quick-start-tutorial) or [try how it works online](https://discoveryjs.github.io/quick-start-tutorial/).

Follow [@js_discovery](https://twitter.com/js_discovery) on Twitter and stay tuned!