Lighthouse architecture demystified
Tags: lighthouse, performance
Published: 2019-03-20 07:57:08

---
title: Lighthouse architecture demystified
published: true
tags: lighthouse, performance
canonical_url: https://izifortune.com/lighthouse-architecture-demystified/
---

### Intro
 ![Lighthouse architecture demystified](http://izifortune.com/content/images/2019/03/lighthouse.jpg)

In one of my previous [articles](https://dev.to/izifortune/performance-budgets-with-lighthouse---lighthouse-keeper-2475-temp-slug-6276913) I explained how we can use _Lighthouse_ to put our website on a budget and why monitoring your website performance its an important aspect of web development. In this article, we will go deeper into _Lighthouse_ building blocks, its architecture and learn how we can start auditing and collecting custom metrics for our web pages.

_Lighthouse_ is an audit tool developed by Google which collects different metrics of your website. After collecting the metrics it will present a series of scores for your webpage. The scoring is divided into five main auditing areas.

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/audit1-2.png)

Besides the scoring of your webpage, _Lighthouse_ provides more detailed information on where to focus your efforts, what is called "opportunities". Areas of your website which impact may lead to big performance improvement as an example the following application execution time are quite high.

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/opportunities-1.png)

### Lighthouse architecture

_Lighthouse_[architecture](https://github.com/GoogleChrome/lighthouse/blob/master/docs/architecture.md#components--terminology) is built around [Chrome Debugging Protocol](https://developer.chrome.com/devtools/docs/debugger-protocol) which is a set of low-level API to interact with a Chrome instance. It interfaces a Chrome instance through the _Driver_. The _Gatherers_ collect data from the page using the _Driver_. The output of a _Gatherer_ is an _Artifact_, a collection of grouped metrics. An _Artifact_ then is used by an _Audit_ to test for a metric. The _Audits_ asserts and assign a score to a specific metric. The output of an _Audit_ is used to generate the lighthouse report that we are familiar with.

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/lighthouse-architcture.png)

We will now take a close look at two of the _Lighthouse_ building blocks by creating a simple audit tracking the internal rendering of a webpage. Creating an application is necessary in order to include a custom _Gatherer_ or _Audit_ since it is not possible to add any custom _Gatherer or Audit_ directly into the _Chrome_ panel.

Let's create our project and install lighthouse as a dependency

```
mkdir custom-audit && cd custom-audit  
npm i --save lighthouse
```

To start auditing our website we will then create a new file `scan.js` where we will import _Lighthouse_ and start scanning the webpage of choice. We will use programmatic access to _Lighthouse_ by importing it inside our project

```
const lighthouse = require('lighthouse');  
const chromeLauncher = require('chrome-launcher');

async function launchChromeAndRunLighthouse(url, opts, config = null) {  
  const chrome = await chromeLauncher.launch({chromeFlags: opts.chromeFlags});
  opts.port = chrome.port;
  const { lhr } = await lighthouse(url, opts, config);
  await chrome.kill()
  return lhr;
}

const opts = {};

// Usage:
(async () => {
  try {
    const results = await launchChromeAndRunLighthouse('https://izifortune.github.io/lighthouse-custom-gatherer', opts);
    console.log(results);
  } catch (e) {
    console.log(e);
  }
})();
```

If we now try to run our file we should be able to see the results coming from a lighthouse scan in the console:

```
node scan.js
```

Now that we have a project with Lighthouse up and running we can start looking at how a Gatherer works and how we can use it in our project. We will use a webpage that I’ve created for this [demo](https://izifortune.github.io/lighthouse-custom-gatherer). In the page, I’m fetching todo list items from an API and rendering on the page. I’m measuring the action using [PerformanceAPI](https://developer.mozilla.org/en-US/docs/Web/API/Performance) as follows:

```
const getDataFromServer = async () => {  
  performance.mark('start');
  const todos = await getTodos();
  renderTodos(todos);
  performance.mark('end');
  performance.measure('Render todos', 'start', 'end');
  const measure = performance.getEntriesByName('Render todos')[0];
}
```

#### Gatherer

A Gatherer is used by Lighthouse to collect data on the page. In fact, any data that is currently needed to perform the default lighthouse audits is collected through a Gatherer. We can extend the Gatherer base class and start creating custom ones:

```
const { Gatherer } = require('lighthouse');

class MyGatherer extends Gatherer {  
  ...
}
```

The class Gatherer defines three different lifecycle hooks that we can implement in our class:

- **_beforePass_** - called before the navigation to given URL
- **_pass_** - called after the page is loaded and the trace is being recorded
- **_afterPass_** - called after the page is loaded, all the other pass have been executed and a trace is available

A lifecycle hook is expected to return either directly an Artifact or a Promise which resolve to the desired Artifact. Depending on what data are we looking to collect from the Driver and at what time we can use any of the hooks just described.

Let’s now create a custom Gatherer which will collect the measurements from the _PerformanceAPI_. The Gatherer needs then to collect entryType `measure` using a PerformanceObserver. We will proceed to create the file `todos-gatherer.js`

```
'use strict';

const { Gatherer } = require('lighthouse');

function performance() {  
  return new Promise((res) => {
    let logger = (list) => {
      const entries = list.getEntries();
      window.todosPerformance = entries[0].duration
      res(entries[0].duration);
    }
    let observer = new PerformanceObserver(logger);
    observer.observe({ entryTypes: ['measure'], buffered: true });
  });
}

class TodosGatherer extends Gatherer {  
  beforePass(options) {
    const driver = options.driver;
    return driver.evaluateScriptOnNewDocument(`(${performance.toString()})()`)
  }

  afterPass(options) {
    const driver = options.driver;
    return driver.evaluateAsync('window.todosPerformance')
  }
}

module.exports = TodosGatherer;
```

Inside TodosGatherer we are using both the _beforePass_ and _afterPass_ hook to contact the Driver and then execute a javascript function inside the context of the current page returning a promise. Inside the _beforePass_ we are registering a _PerformanceObserver_ just after the page will load, since the observers are not buffered we might encounter in a race condition. In the _afterPass_ then we collect the previously registered measure.   
To get an idea of all the methods that you use on the driver object you can have a look   
[here](https://github.com/GoogleChrome/lighthouse/blob/34a542a156b271fd7725525b09a92a344d7809c0/lighthouse-core/gather/driver.js).

Now we need to include it in our `scan.js` file:

```
const lighthouse = require('lighthouse');  
const chromeLauncher = require('chrome-launcher');

const config = {  
  passes: [{
    passName: 'defaultPass', //Needed to run custom Gatherers/Audits in the same pass
    gatherers: [
      `todos-gatherer`,
    ],
  }],
}
...
```

If we try to run `scan.js` at this moment we will receive an error that there are no audits to run. A Gatherer on its own doesn’t provide any information but rather output Artifacts used on the Audits to define metrics. To proceed then we will have a look at the Audits then.

#### Audit

An Audit defines a metric or score, it takes the Artifacts as an input and calculates the desired score. The different audits that _Lighthouse_ is performing such as _FirstMeaningfulPaint_ or _SpeedIndex_ are all in fact defined as an audit internally.   
To create a custom _Audit_, similar to a _Gatherer_, we will extend a base class `Audit` and implements the basic methods:

To create a custom _Audit_, similar to a _Gatherer_, we will extend a   
base class `Audit` and implements the basic methods:

```
const { Audit } = require('lighthouse');

class MyAudit extends Audit {  
  static get meta() {
    ..
  }

  static audit(artifacts) {
    ...
  }
}
```

The class _Audit_ defines two methods that need to be overridden:

- **_meta_** - used to define information about the audit
- **_audit_** - takes as input the _Artifacts_ from Gatherers and return a Product metric.

With this information in mind, we can now implement our custom Audit and start collecting the performance of the todo list. The name of the custom audit file will be `todos-audit.js` and will contain:

```
'use strict';

const Audit = require('lighthouse').Audit;

class TodosAudit extends Audit {  
  static get meta() {
    return {
      id: 'todos-audit',
      title: 'Todos are loaded and rendered',
      scoreDisplayMode: Audit.SCORING_MODES.NUMERIC,
      failureTitle: 'Todos loading is too slow.',
      description: 'Used to measure time for fetching and rendering todos list',
      requiredArtifacts: ['TodosGatherer'],
    };
  }

  static audit(artifacts) {
    const measure = artifacts.TodosGatherer;

    return {
      rawValue: measure,
      score: Math.max(1 - (measure / 1500), 0),
      displayValue: `Todos rendering is: ${measure}ms`
    };
  }
}
module.exports = TodosAudit;
```

Inside the method `meta` we are defining information describing the _Audit_ itself such as id, title, scoreDisplayMode and description. Also, we are configuring the _Artifacts_ which are needed by the _Audit_ in this case `TodosGatherer` is the name of the Gatherer of interest.

And now we need to add it in the configuration inside `scan.js` similar to what we did previously to the _Gatherer_.

```
const config = {  
  passes: [{
    passName: 'defaultPass’,
    gatherers: [
      `todos-gatherer`,
    ],
  }],
  audits: [
    'todos-audit',
  ],
  categories: {
    todos: {
      title: 'Todos metrics',
      description: 'Performance metrics for todos',
      auditRefs: [
      // When we add more custom audits, `weight` controls how they're averaged together.
      {id: 'todos-audit', weight: 1},
    ],
    },
  },
}
```

Launching now our scan we can notice our custom audit being logged into the console.

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/console.png)

If you prefer to have the report in a different format such as HTML you can add the output option to the lighthouse function. The options object will be then used by _Lighthouse_ to configure the running audits output format. To recap the final `scan.js` will look like:

```
const lighthouse = require('lighthouse');  
const chromeLauncher = require('chrome-launcher');  
const { promisify } = require('util');  
const { writeFile } = require('fs');  
const pWriteFile = promisify(writeFile);

const config = {  
  passes: [{
    passName: 'defaultPass’,
    gatherers: [
      `todos-gatherer`,
    ],
  }],
  audits: [
    'todos-audit',
  ],
  categories: {
    todos: {
      title: 'Todos metrics',
      description: 'Performance metrics for todos',
      auditRefs: [
      // When we add more custom audits, `weight` controls how they're averaged together.
      {id: 'todos-audit', weight: 1},
    ],
    },
  },
}

async function launchChromeAndRunLighthouse(url, opts, config = null) {  
  const chrome = await chromeLauncher.launch({chromeFlags: opts.chromeFlags});
  opts.port = chrome.port;
  const { lhr, report } = await lighthouse(url, opts, config);
  await chrome.kill()
  return report;
}

const opts = {  
  output: 'html'
};

// Usage:
(async () => {
  try {
    const results = await launchChromeAndRunLighthouse('https://izifortune.github.io/lighthouse-custom-gatherer', opts, config);
    await pWriteFile('report.html', results)
  } catch (e) {
    console.log(e);
  }
})();
```

And running it now we will have an HTML report inside `report.html` which will look similar to the following one:

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/report1.png)

We can include also the standards _Lighthouse_ audits together with our custom one by adding to the configuration object the following key:

```
const config = {  
  extends: 'lighthouse:default', // Include Lighthouse default audits
  passes: [{
    passName: 'defaultPass’,
    gatherers: [
      `todos-gatherer`,
    ],
  }],
...
```

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/full-report.png)

After the introduction to the Lighthouse architecture, we can start customising the Lighthouse audits by measuring and reporting metrics which are relevant to us. We will explore now how we can use the Gatherers to overcome a common problem while performing the scans on a CI environment.

### Session guard pages

In _Ryanair_, we are using _Lighthouse_ extensively to audit our webpages as part of an automated job performing scans at regular intervals and then we analyse the results of the scans on a regular basis. One of the main problems that we encountered when you are running an automated scan is how to perform audits on pages behind an authentication or user session. While with manual scans we can easily generate a session before starting the audit; if we are running _Lighthouse_ from a CI environment we will need to generate a session programmatically and pass the information to _Lighthouse._

A common approach for user session authentications management for a web application is to generate tokens, often JWT, on the server after a successful login and store the result token in the browser. You can store the token in different storage available in the browser such as _LocalStorage, SessionStorage, Cookies._ I will not judge here where is the best place to store a token what is interesting to us is how we can write to any of that browser storage so that _Lighthouse_ can access the token and perform an audit.

By using a custom _Gatherer_ we can create a user session by leveraging the lifecycle hook _beforePass_ which triggers before the navigation to the page URL. In the hook, we call an API to generate a session and then through one of the _Driver_ methods `evaluateScriptOnNewDocument` we can pass any function to be executed in the browser instance.

For the purpose of this demo, I've created another [page](https://izifortune.github.io/lighthouse-custom-gatherer/auth) where I'm basically rendering the todos only if the user is authenticated. To fake the authentication I'm checking that a specific token is present in _LocalStorage_ and then start fetching and rendering todos.

Let's create a new _Gatherer_ called _SessionGatherer_ in the file   
`session-gatherer.js`

```
const { Gatherer } = require('lighthouse');

const TOKEN = 'iOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImp0aSI6IjkzZDU0MDBiLTQ5MzgtNDNmZS1iMjY4LTY2MDJlNDIxMjFiYiIsImlhdCI6MTU1MjkwNjc0NywiZXhwIjoxNTUyOTEwMzQ3fQ.qEJflkN2ntXrQFalBkkw4duCh55HdNBLGXZOV-dS3KQ';

function createSession(token) {  
  localStorage.setItem('token', token);
}

class SessionGatherer extends Gatherer {

  async beforePass(options) {
    const driver = options.driver;
    return driver.evaluateScriptOnNewDocument(`(${createSession.toString()})('${TOKEN}')`);
  }
}

module.exports = SessionGatherer;
```

Once we have created the gatherer we need to tell _Lighthouse_ to include it in the list so that it will be running alongside all the other gatherers while performing an audit. We need to will create another file `scan-auth.js` as following:

```
const lighthouse = require('lighthouse');  
const chromeLauncher = require('chrome-launcher');  
const { promisify } = require('util');  
const { writeFile } = require('fs');  
const pWriteFile = promisify(writeFile);

const config = {  
  extends: 'lighthouse:default',
  passes: [{
    passName: 'defaultPass',
    gatherers: [
      `session-gatherer`,
      `todos-gatherer`
    ],
  }],
  audits: [
    'todos-audit'
  ],
  categories: {
    todos: {
      title: 'Todos metrics',
      description: 'Performance metrics for todos',
      auditRefs: [
      // When we add more custom audits, `weight` controls how they're averaged together.
      {id: 'todos-audit', weight: 1},
    ],
    },
  },
}

async function launchChromeAndRunLighthouse(url, opts, config = null) {  
  const chrome = await chromeLauncher.launch({chromeFlags: opts.chromeFlags});
  opts.port = chrome.port;
  const { lhr, report } = await lighthouse(url, opts, config);
  await chrome.kill()
  return report;
}

const opts = {  
  output: 'html'
};

// Usage:
(async () => {
  try {
    const results = await launchChromeAndRunLighthouse('https://izifortune.github.io/lighthouse-custom-gatherer/auth', opts, config);
    await pWriteFile('report.html', results)
  } catch (e) {
    console.log(e);
  }
})();
```

Now we can start running our scans on the pages behind a user session and monitor their performance.

```
node scan-auth.js
```

Which record the default _Lighthouse_ metrics plus the custom Todos metrics which on this case are behind a user authentication.

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/full-report-auth.png)

Inside the report now if you look at the _filmstrip_ you will notice that the todo list got rendered correctly meaning that a valid token was found in _LocalStorage_.

![Lighthouse architecture demystified](https://izifortune.com/content/images/2019/03/filmstrip.png)

I've collected the examples presented here in this repository [https://github.com/izifortune/lighthouse-custom-gatherer](https://github.com/izifortune/lighthouse-custom-gatherer)

### Credits

The initial idea came by looking at the custom audit recipe from the _Lighthouse_ team that you can find [here](https://github.com/GoogleChrome/lighthouse/tree/master/docs/recipes/custom-audit).   
The recipe served as an inspiration and a first example to create a custom Gatherer/Audit. I also would like to thanks @patrickhulce for his time and prompt answers on Gitter.