How we wrote our CLI integration tests
Tags: javascript, webdev, devops, node
Published: 2022-03-25 14:29:59

*Cover image from Unsplash by [Glenn Carstens-Peters](https://unsplash.com/@glenncarstenspeters)*

One of the most important parts of software is ensuring that it works - not only on your machine, but also on the target machines.

The more variables are there the more complex it is to create reliable software. What seems easy first quickly becomes a mess of checking edge cases and identifying scenarios.

For the command line tooling of [our micro frontend framework Piral](https://github.com/smapiot/piral) we needed to be sure that it properly runs. This includes

- testing against different operating systems (Windows, Linux, Mac)
- testing against different versions of Node.js (starting with 12)
- testing against different bundlers (most importantly Webpack, but also Parcel, esbuild, vite, ...)

All in all not an easy task. While we have quite a high (90+) percentage of unit test coverage, experience has taught us that nothing can replace integration tests. This is the only way to identify issues with underlying operating systems or runtimes.

![Testing](https://media1.giphy.com/media/7MZ0v9KynmiSA/giphy.gif)

Let's see what we did to run our tests.

## The basic setup

Our tests will run in the command line using a tool set consisting of

- Jest (test runner)
- Playwright (to check if debugging / build artifacts work properly) together with `expect-playwright` for simplified assertions
- TypeScript (to make sure the test code base itself does not contain some easy mistake)
- Azure Pipelines (running the tests in different environments)

The [code for our CLI integration tests](https://github.com/smapiot/piral-cli-integration-tests) is on GitHub.

The setup of Jest (done via the `jest.config.js` file) can be broken down to the following:

```js
const { resolve } = require('path');

const outDirName = process.env.OUTDIR || 'dist';
const outputDirectory = resolve(process.cwd(), outDirName);

process.env.OUTPUT_DIR = outputDirectory;

module.exports = {
  collectCoverage: false,
  globals: {
    NODE_ENV: 'test',
    'ts-jest': {
      diagnostics: false,
    },
  },
  testEnvironmentOptions: {
    'jest-playwright': {
      browsers: ['chromium'],
      exitOnPageError: false,
      collectCoverage: false,
      launchOptions: {
        headless: true,
      },
    },
  },
  setupFilesAfterEnv: ['expect-playwright'],
  testTimeout: 2 * 60 * 1000,
  preset: 'jest-playwright-preset',
  reporters: [
    'default',
    [
      'jest-junit',
      {
        outputDirectory,
      },
    ],
  ],
  transformIgnorePatterns: [
    '<rootDir>/node_modules/',
    'node_modules/@babel',
    'node_modules/@jest',
    'signal-exit',
    'is-typedarray',
  ],
  testPathIgnorePatterns: ['<rootDir>/node_modules/'],
  modulePathIgnorePatterns: ['<rootDir>/node_modules/'],
  roots: ['<rootDir>/src/'],
  testRegex: '(/__tests__/.*|\\.test)\\.ts$',
  testURL: 'http://localhost',
  transform: {
    '^.+\\.ts$': 'ts-jest',
    '^.+\\.js$': 'babel-jest',
  },
  moduleFileExtensions: ['ts', 'js', 'json'],
  moduleNameMapper: {},
  verbose: true,
};
```

While some parts, e.g., the integration of `ts-jest` for TypeScript support, are rather straight forward, other parts are not. Especially the `transformIgnorePatterns` and `testEnvironmentOptions` require some explanation.

The `transformIgnorePatterns` (along the `testPathIgnorePatterns` and `modulePathIgnorePatterns`) are necessary to actually support the use case of providing the tests via an npm package (i.e., as a library). This use case is necessary to make the tests also available to other bundler plugins, which are not already covered by running the tests within the repository. We'll go into details later.

The `testEnvironmentOptions` enable the use of Playwright. Playwright is a browser automation tool that helps us controlling a browser, e.g., to check if certain elements are also rendered. This is necessary for some tests to actually verify that everything was done right.

## Matrix testing

To run the tests in different environments we use a CI/CD feature called matrix strategy. This will run the same pipeline in different variations.

```yaml
strategy:
  matrix:
    linux_node_12:
      imageName: "ubuntu-20.04"
      nodeVersion: 12.x
    linux_node_14:
      imageName: "ubuntu-20.04"
      nodeVersion: 14.x
    linux_node_16:
      imageName: "ubuntu-20.04"
      nodeVersion: 16.x
    linux_node_17:
      imageName: "ubuntu-20.04"
      nodeVersion: 17.x
    windows_node_14:
      imageName: "windows-2019"
      nodeVersion: 14.x
    macos_node_14:
      imageName: "macOS-11"
      nodeVersion: 14.x
```

Whenever we have a new environment to be tested we just add it here. All the rest, e.g., what base image is selected to run the pipeline is then determined using the variables from the matrix.

The remaining steps in the CI/CD pipeline are then rather straight forward:

```yaml
steps:
  - task: NodeTool@0
    inputs:
      versionSpec: $(nodeVersion)
    displayName: "Install Node.js"

  - script: npm install --legacy-peer-deps
    displayName: "Setup Tests"

  - script: npm test
    continueOnError: true
    displayName: "Run Tests"
    env:
      CLI_VERSION: ${{ parameters.piralCliVersion }}

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: "JUnit"
      testResultsFiles: "dist/junit*.xml"
      mergeTestResults: true
```

We first change to the selected version of Node.js and then prepare running the tests by installing all dependencies. Then - and this is the most important step - we actually run the tests. We pass in version of the CLI we actually want to test. By default, this is set to the `next` tag on [npm of the `piral-cli` package](https://www.npmjs.com/package/piral-cli).

We could also run the tests for a different version. All we would need to do is pass a different value for this parameter when starting the pipeline.

Finally, we publish the test results. We use the package `jest-junit` to store the results in the JUnit format, which is compatible with the `PublishTestResults@2` task of Azure Pipelines.

## Code structure and utilities

The code contains three directories:

- `bin` has a small wrapper that can be used to run the tests as an npm package
- `src` contains all the tests
- `src/utils` contains the utilities to efficiently write the tests

The utilities make it possible to conveniently provide integration tests for our CLI tool. These utilities can be categorized:

- context / jest enhancing
- convenience for input / output handling
- dealing with processes (starting, stopping, monitoring, ...)
- running a server to emulate CLI to service interaction

While standard Jest unit tests look a bit like

```js
import someFunction from './module';

describe('Testing module', () => {
  it('works', () => {
    // arrange
    const input = 'foo';
    // act
    const output = someFunction(input);
    // assert
    expect(output).toBe('bar');
  });
});
```

the tests in this repository look a bit different:

```js
import { runTests } from './utils';

runTests('cli-command', ({ test, setup }) => {
  // "common" arrange
  setup(async (ctx) => {
    await ctx.run(`npm init -y`);
  });

  test('some-id', 'works', ['feature'], async (ctx) => {
    // act
    await ctx.run('npm test');

    // assert
    await ctx.assertFiles({
      'coverage/coverage-final.json': true,
    });
  });
});
```

First of all, there are no modules or functions to import for testing here. We only import utilities. The most important utility is the `runTests` wrapper. This will give us access to further (specialized) wrappers such as `setup` and `test`. The former is a generic arrange. Everything that is run in there, will produce content that can be used (i.e., will be present) for each test.

Since some commands may install packages or perform longer operations (in the area of 10 to 40 seconds) it is crucial to not run the actual common arrange steps again. Instead, it is assumed that there are some outputs to the context directory, which can then be just copied from a temporary arrange location to the temporary test location.

![Efficiency](https://media3.giphy.com/media/sa0LyqJW402YM/giphy.gif)

The ability to conveniently have a temporary directory underneath (where everything else is relative to) is the reason for having these wrappers such as `runTests`, `setup`, or `test`.

The basic flow here is:

1. For a test suite create a "container" directory in a predefined output directory (usually `dist`)
2. Run the setup steps (once for all tests in a test suite) in a dedicated "template" directory inside the container directory
3. Run the tests, each test creates its own temporary directory inside the container directory
4. For each test first copy the contents of the template directory to it

That way, the outcome can be easily inspected and removed. Otherwise, finding the outcome - or cleaning it up - becomes a mess.

To find individual tests easier the directory of each test is prefixed with the id (in the example above `some-id`) that we give it. It also contains a random string to make sure there are no collisions.

## Running the tests

Let's look at one of the more complicated tests:

```js
import axios from 'axios';
import { cliVersion, runTests, selectedBundler, getFreePort } from './utils';

runTests('pilet-debug', ({ test, setup }) => {
  setup(async (ctx) => {
    await ctx.run(`npx --package piral-cli@${cliVersion} pilet new sample-piral@${cliVersion} --bundler none`);
    await ctx.run(`npm i ${selectedBundler} --save-dev`);
  });

  // ...

  test(
    'debug-standard-template-with-schema-v0',
    'can produce a debug build with schema v0',
    ['debug.pilet'],
    async (ctx) => {
      const port = await getFreePort(1256);
      const cp = ctx.runAsync(`npx pilet debug --port ${port} --schema v0`);

      await cp.waitUntil('Ready', 'The bundling process failed');

      await page.goto(`http://localhost:${port}`);

      const res = await axios.get(`http://localhost:${port}/$pilet-api`);
      const pilets = res.data;

      expect(pilets).toEqual({
        name: expect.anything(),
        version: expect.anything(),
        link: expect.anything(),
        spec: 'v0',
        hash: expect.anything(),
        noCache: expect.anything(),
      });

      await expect(page).toHaveSelectorCount('.pi-tile', 1);

      await expect(page).toMatchText('.pi-tile', 'Welcome to Piral!');
    },
  );
});
```

Here we set up a micro frontend (called a "pilet") using `npx` with the `piral-cli` command. Then we install the selected bundler to be able to verify the `debug` command.

To prevent potential conflicts on the used port we use an utility for finding the next free port (default: 1256). Then we start an ever-running command `npx pilet debug`. Unlike the simple `run` the `runAsync` will run concurrently by default. Still, we want to wait until the command would print "Ready" in the console. If we find something like "The bundling process failed" (or even a terminated application) then we will have a failed test.

After the debug process is ready we can finally use Playwright to go to the page and run some assertions. We check against the debug server if it contains the expected API response.

Furthermore, we can assertions on the website. We should find a tile on the dashboard coming from the micro frontend that we are currently debugging.

So how can we run it? We can run it from the command line using `npm start`. If we want to run a specific test, e.g., for the `pilet debug` command we can also run `jest` directly.

```sh
npx jest src/pilet-debug.test.ts
```

Theoretically, we could also run a specific test:

```sh
npx jest src/pilet-debug.test.ts -t 'can produce a debug build with schema v0'
```

This works in almost all test suites except the ones using Playwright. In those test suites the `page` object remains undefined as some "magic" that is performed by the Jest Playwright integration is not present in such a scenario.

Besides running (all) the tests from the test repository the tests could also be installed and run locally:

```sh
npm i @smapiot/piral-cli-integration-tests
npx piral-cli-tests
```

Note that this will not run all tests, but only the tests that require a bundler. Using this way one could test a self-developed bundler plugin. In the future, this would also provide the whole CLI test infrastructure to quickly allow testing other `piral-cli` plugins, too.

## Results

Right now the tests run on-demand, even though they could (for whatever reason) also be scheduled. Already when writing the tests we've detected some edge cases and little improvements that helped us make the `piral-cli` even better.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6x38je5b47z9b2cnvk7s.png)

So overall, besides having the assurance with new releases that we did not unintentionally break something, we already gained quite a bit of value from having integration tests in that area.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u8k50xbsx206gz65drlo.png)

Right now failing tests are essentially reported as "partially failing" as we continue in the pipeline to actually publish them.