Fast and reliable end-to-end tests with Playwright on GitHub Actions
Tags: webdev, github, devops, testing
Published: 2023-01-24 17:48:30

[Playwright](https://playwright.dev/) is a powerful web testing tool supporting Chromium, Firefox and WebKit engines. The project has excellent documentation and many examples.

I use Playwright for E2E tests on the DEV environment. Today I'm going to show how you can integrate Playwright on GitHub Actions more efficiently.


![Playwright by Microsoft](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1mk7qxx3ifxc5hdi66zl.png)

The documentation [suggests](https://playwright.dev/docs/ci#github-actions) using the following snippet to run Playwright inside GitHub Actions:
```yml
steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-node@v3
    with:
      node-version: '18'
  - name: Install dependencies
    run: npm ci
  - name: Install Playwright
    run: npx playwright install --with-deps
  - name: Run your tests
    run: npx playwright test
  - name: Upload test results
    if: always()
    uses: actions/upload-artifact@v3
    with:
      name: playwright-report
      path: playwright-report
```

What is notable here?
* It's supposed to run all tests on the same type of GitHub Runner. `ubuntu-latest` by default.
* All supported browser engines (chromium, firefox, WebKit) will be installed on the Linux runner.
* Playwright will execute tests for all browser engines from the same runner.
* The last step will prepare an archive with an HTML report.

After several months of experiments I come up with the following configuration:
```yml
# ...

jobs:
  test:
    name: 🧪 ${{ matrix.project }} E2E Tests
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        include:
          - project: chromium
            os: ubuntu-latest
            cache_dir: ~/.cache/ms-playwright
          - project: firefox
            os: ubuntu-latest
            cache_dir: ~/.cache/ms-playwright
          - project: webkit
            os: macos-12
            cache_dir: ~/Library/Caches/ms-playwright
    steps:

    - uses: actions/checkout@v3

    - uses: actions/setup-node@v3
      with:
        node-version-file: '.nvmrc'
        cache: 'npm'

    - name: ⚡️ Cache NPM dependencies
      uses: actions/cache@v3
      id: cache-primes
      with:
        path: node_modules
        key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}

    - name: ⚡️ Cache playwright binaries
      uses: actions/cache@v3
      id: playwright-cache
      with:
        path: ${{ matrix.cache_dir }}
        key: ${{ runner.os }}-${{ matrix.project }}-pw-${{ hashFiles('**/.playwright-version') }}

    - name: 📥 Install deps
      if: steps.cache-primes.outputs.cache-hit != 'true'
      run: make install

    - name: 📥 Install ${{ matrix.project }} with Playwright
      if: steps.playwright-cache.outputs.cache-hit != 'true'
      run: npx playwright install --with-deps ${{ matrix.project }}

    - name: 🎭 Playwright tests
      run: npx playwright test --project=${{ matrix.project }}
      env:
        DEBUG: pw:api,pw:browser*

    - name: 📊 Upload test results
      if: failure()
      uses: actions/upload-artifact@v3
      with:
        name: playwright-report-${{ matrix.project }}
        path: playwright-report
```

What I've changed in GitHub Workflow for Playwright tests?
* I use a matrix with three browser engines to run the jobs simultaneously. Instead of `ubuntu-latest`, I run WebKit tests on `macos-12` runner. I had to make the change to avoid fragile tests. I noticed that the `page.goto()` API works differently on Linux and macOS for the WebKit engine. WebKit didn't wait for `load` event, which causes failures from time to time. At the same time, I've never caught such a problem locally on my MacBook. If you're interested, [the ticket](https://github.com/microsoft/playwright/issues/8340) is still open in the Playwright bug tracker.
* The `fail-fast` parameter is set to false to wait for all testing jobs to complete, even if one of them fails for some reason.
* I believe it is essential to specify `timeout-minutes` for E2E tests so that the tests will crash if the infrastructure hangs or behaves incorrectly.
* I use the _.nvmrc_ file inside the repository to specify the NodeJS version. DRY all over the place!
* [**Configured aggressive cache**](https://dev.to/drakulavich/aggressive-dependency-caching-in-github-actions-3c64) for npm dependencies and binary files with browsers. I added a special target to Makefile to calculate the Playwright browsers hash:
```make
bump-playwright:
  npm install --save-dev @playwright/test
  npx playwright --version > .github/.playwright-version
```

After executing `make bump-playwright` command, the latest PW version will be installed and the value of the installed release will be written to the _.github/.playwright-version_ file. GitHub Actions calculate the hash for the cache from the file content. Example file:
```sh
$ cat .github/.playwright-version
Version 1.29.2
```

* Each runner installs only one browser and it's pretty handy cause the HTML report will have the results for that browser — easy to navigate and spot the issues. This distinction also allows you to have an idea of the test execution speed in each browser and to notice flickering tests specific to a particular engine.
* Playwright tests run with enabled DEBUG, so that in case of a crash, you can quickly get additional information for diagnostics.
* In contrast to the recommended configuration from the documentation, I prefer to upload artifacts with the report only in case of test failures. I don't see the point in storing those artifacts on the cloud, given how often the tests are run.

I shared my findings for configuring E2E playwright tests in GitHub Actions. Feel free to ask questions about the example GitHub workflow.

What Playwright tricks do you use in your daily work? Would love to hear about your hidden gems 💎.

Cover photo by [Denny Müller](https://unsplash.com/photos/JySoEnr-eOg).