Visual regression tests for Storybook with Loki
Tags: visualregressionstests, javascript, storybook, designsystem
Published: 2023-02-22 10:58:49

---
title: 'Visual regression tests for Storybook with Loki'
published: true
tags:
  - visualregressionstests
  - javascript
  - storybook
  - designsystem
canonical_url: https://grischuk.de/posts/storybook-visual-regressions-tests-with-loki
---

How to setup visual regression tests with Loki for your design system

To my knowledge, there are currently no leading standalone tools for visual regression testing. Many companies implement their own solutions, but there are no tools that are widely used and recommended as a standard.

Some companies actively explore OSS solutions and in this article, I will share my experience with one of them.

## What is Loki?

[Loki](https://loki.js.org/) is a tool that makes it easy to test Storybook projects for visual regressions. Loki was used for the trial stage at [Thoughtworks tech radar](https://www.thoughtworks.com/en-de/radar).

## Why Loki?

> There are a few visual regression tools for the web, but most either cannot be run headless or use `phantomjs` which is deprecated and a browser nobody is actually using. They usually also require you to maintain fixtures. With react-native it's now possible to target multiple platforms with a single codebase, but there's no single tool to test all to my knowledge.


## How to setup Loki for your project?

### 1. Install Loki package

```bash
npm i loki --save-dev
```

or if you use yarn

```bash
yarn add loki -D
```

### 2. Create a Loki config file

```bash
yarn loki init
```

This will add standard configuration to your project. You can find it in the `package.json` file.

Example of the configuration:

```json
{
  "name": "my-project",
  "version": "1.0.0",
  "loki": {
    "chromeSelector": "#my-decorator > *",
    "configurations": {
      "chrome.laptop": {
        "target": "chrome.app",
        "width": 1366,
        "height": 768
      },
      "chrome.iphone7": {
        "target": "chrome.docker",
        "preset": "iPhone 7"
      },
      "chrome.a4": {
        "target": "chrome.aws-lambda",
        "preset": "A4 Paper"
      },
      "ios": {
        "target": "ios.simulator"
      },
      "android": {
        "target": "android.emulator"
      }
    }
  }
}
```

I prefer not to pollute my `package.json` and extract configuration to a separate file `loki.config.js`:

```javascript
// loki.config.js
module.exports = {
  diffingEngine: 'pixelmatch',
  configurations: {
    'chrome.laptop': {
      target: 'chrome.docker',
      width: 1366,
      height: 768,
      deviceScaleFactor: 1,
      mobile: false,
    },
    // 'chrome.iphone7': {
    //   target: 'chrome.docker',
    //   preset: 'iPhone 7'
    // }
  },
}
```

I'll go with the simplest configuration for now, but you can find more information about the configuration options in the [Loki documentation](https://loki.js.org/configuration.html).

I choose the `pixelmatch` engine for now, but you can also choose between [GraphicsMagick](http://www.graphicsmagick.org/) (`gm`) and `looks-same`. Currently, three diffing engines are available. Worth to mention that if you'd like to use `graphicsmagick` diffing engine,
for image comparison, you need to install it on the machine / CI as an additional dependency.

### 3. Install Docker

Loki uses Docker to run tests in an image. If you don't have Docker installed, you can follow the [installation instructions](https://docs.docker.com/install/).

### 4. Start Storybook

When running tests locally, you need to run a local instance of the Storybook & start Docker to execute the tests.

```bash
yarn storybook
```

### 5. Generate reference images

```bash
yarn loki update
```

will create a reference image for each story in your Storybook & place them in the `.loki/reference` directory.

### 6. Run tests

```bash
yarn loki test
```

or if you want to run tests for a specific platform (e.g. laptop, iphone7 etc.)

```bash
yarn loki test laptop
```

On each test run, Loki will create new files in the `.loki` directory. Inside this directory, you will find the `current`, `difference`, and `reference` folders.

- The `current` directory contains the images that were generated during the test run.

- The `difference` directory contains the images that were generated during the test run and are the ones that were different from the reference images with, visual differences shown.

- The `reference` directory contains the reference images that were generated during the `loki update` command. These are the images that are used to compare against the current images.

Images from the `reference` directory should be committed to the repository if you want to use them as a reference for your tests on CI.

### 7. Add Loki to CI (as GitHub action)

To run Loki on CI you need to add a new job to your CI pipeline.

Update NPM scripts

```json
{
  "scripts": {
    "visual-tests": "loki test laptop",
    "visual-tests-ci": "build-storybook && loki --requireReference --reactUri file:./storybook-static"
  }
}
```

Add a new job to your CI pipeline as a GitHub action

```yaml
# .github/workflows/ci.yml

visual-regression-tests:
  name: 'Visual tests: React DOM'
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v3
    - name: Setup node from node version file
      uses: actions/setup-node@v3
      with:
        node-version: '16'
        cache: 'yarn'
        registry-url: 'https://npm.pkg.github.com/'
        scope: '@your-org'
    - name: Install dependencies
      run: yarn --frozen-lockfile --non-interactive --silent --ignore-scripts
      env:
        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
    - name: Run loki
      run: yarn visual-tests-ci
    - name: Archive screenshots
      if: ${{ failure() }}
      uses: actions/upload-artifact@v1
      with:
        name: design-system
        path: .loki
```

#### Skipping stories

If you want to skip some stories from the tests you can add `loki.skip` parameter to the story:

```jsx
<!-- Badge.stories.mdx -->

import { Canvas, Story } from '@storybook/addon-docs'
import Badge from './Badge'

<Canvas>
  <Story
    name="warning"
    parameters={{
      loki: {
        skip: true
      }
    }}
  >
    <Badge variant="warning">
      On hold
    </Badge>
  </Story>
</Canvas>
```

#### Approve changes

If you want to approve changes you can run:

```bash
yarn loki approve
```

This will update images in the reference folder if changes are intended and valid.

#### Accessing artifact files from GitHub action

In case of a flaky test or a test that fails on CI but passes locally, you can download the artifact files and run the tests locally to see what's going on.

To access artifact files you can go to the "Actions" tab in your repository, find your workflow build, navigate to "Summary" and download the artifact from the latest run:


![design-system-artifacts](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q349ii4og1kc93lmz7ha.png)

### Pros

- Relatively easy to setup
- Multi devices/platforms support
- Running in Docker
- Different comparison engines to choose from

### Cons

- Building a Storybook on each test run, Loki requires it to perform tests on a static build output. The build takes time & resources on CI. A possible solution: cache this somehow
- No handy way to visually compare/accept differences/changes between the test runs

## Comparison stats - VRT tests

Tests were run on a local machine and on a CI machine (build time excluded).

The tests were run on a Storybook with 30 components and on average 3 stories per component.

The tests were run with 3 different comparison engines.

There are _three_ currently available options to choose from when comparing images in Loki.

| Tests runs | [pixelmatch](https://github.com/mapbox/pixelmatch) | [gm ](https://github.com/aheckmann/gm) | [looks-same](https://github.com/gemini-testing/looks-same) |
| ---------- | -------------------------------------------------- | -------------------------------------- | ---------------------------------------------------------- |
| Locally (M1)    | ~27.71s ✅                                   | ~27.7s ✅                        | ~29.09s ✅         


## Conclusion

It's important to have a visual regression testing tool in your tool belt. It's a great way to catch visual bugs early and prevent them from reaching production.

There are many things to consider when implementing visual regression testing, it should not slow down your development process and release process.

Although there are some cons to using Loki, I'd give it a try and see if it works for you. It's a decent tool for catching visual bugs and it's relatively easy to set up.

If you have any questions or suggestions, feel free to reach out to me on Twitter [@alxgri](https://twitter.com/alxgri)