Using GitHub actions and Vercel for end-to-end tests
Tags: github, e2e, testing, vercel
Published: 2021-06-04 14:22:10

[End-to-end](https://www.katalon.com/resources-center/blog/end-to-end-e2e-testing/) (E2E) tests are the tip of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html).
They are supposedly the hardest to write and take the longest to run.
But they are also valuable as they are the tests that "use" your app as your users do.

When we first started to use E2E tests at [Signavio](https://www.signavio.com/), we were able to run them after changes got merged to our `master` branch but not on [feature branches](https://www.atlassian.com/de/git/tutorials/comparing-workflows/feature-branch-workflow).
That is problematic because it had already made it to our mainline and possibly to production when they uncovered a defect.
Another obstacle to not run them on each [PR](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) was that spinning up an environment was not an easy task and would take considerable time.
Since each minute the pipeline runs prolongs the feedback loop for the individual developer, we had a problem.

When [GitHub](http://github.com/) launched [actions](https://github.com/features/actions) we discovered that actions can run when a [deployment](https://docs.github.com/en/actions/reference/events-that-trigger-workflows#deployment) happens.
By then, we already had set up [Vercel](http://vercel.com/) to deploy our client on each PR.
I asked myself whether I could connect the Vercel deployments with our E2E tests.

## TL;DR

No, shortlist this time.
Sorry! Suppose you are solely interested in connecting [cypress](https://www.cypress.io/) to a Vercel deployment.
In that case, you can skip ahead to the [Creating the workflow](#creating-the-workflow) section.
If this is the first action you write, then I would recommend the full article.
If you don't have time for that, definitively read [Caveats](#caveats).
That might save you some time.

## Vercel deployments

Setting up your project in Vercel is (and I don't say this lightly) easy if you're working with solutions like [create react app](https://github.com/facebook/create-react-app) or static site generators like [Gatsby](http://gatsbyjs.org/).
Vercel can import and run your project without you needing to do any custom configuration.
But even if you didn't bootstrap your page like that, the configuration [is straightforward](https://vercel.com/docs/configuration#project/builds).

Another reason I'm pointing at Vercel is that they already integrate with GitHub and register themselves as a deployment which is essential for the next step.

## GitHub actions

GitHub actions are a powerful tool.
But I also find them a bit hard to get started with.
I struggled a lot with wording and how to achieve a simple goal.
This article is in no form a complete guide to writing good actions.
It's merely a description of what I needed to do to get the job done.

Let's write an action! Wrong.
The first thing I had to learn was that what I needed to do is add a **workflow**.
I got fooled by the "Actions" tab on GitHub and assumed it would list actions.
But what you will see there are workflows.

An action is one **step** in a workflow.
A workflow composes actions so that you can achieve a more complex goal.
You also need the workflow to express when you want these actions to run.
Workflows are co-located with the code inside one repository.
For GitHub to run a workflow, you need to create a [`YAML`](https://en.wikipedia.org/wiki/YAML) file inside the `.github/workflows/` folder.

```yaml
name: E2E tests
on: [deployment_status]
```

Why does this say `deployment_status` and not `deployment`?Because we wanted to improve the _developer_ experience.
To give our developers fast feedback on their PRs, we set a custom status.
For that, we need some more information about the deployments.

Suppose you are not interested in building a custom action.
In that case, you can skip ahead to [Creating the workflow](#creating-the-workflow).

As we will need to work with the GitHub API to set a status on a PR, we will now create a custom action for that.
GitHub offers [two ways to define actions](https://docs.github.com/en/actions/creating-actions/about-actions#types-of-actions) - JavaScript and [Docker](https://www.docker.com/).
In this example, we will create the action using Docker since this is how I know to do it.
We will also use the [`actions-toolkit`](https://github.com/actions/toolkit) to make building custom actions easier.

### An action to set a PR status

Since we do not intend to release this action for other developers, we can put all code into a `.github/actions/set-pr-status` folder inside our repository.
This folder will contain three files.

- a `package.json` because we want to use the `actions-toolkit` package,
- a `DOCKERFILE` that describes the container to run our code, and
- an `action.js` file that contains the code of our action

Let's start with the `package.json.`
Since this will not be a package you want to release on [NPM](https://www.npmjs.com/), we can keep it short.

```json
{
  "name": "set-pr-status",
  "private": true,
  "main": "action.js",
  "dependencies": {
    "actions-toolkit": "5.0.0"
  }
}
```

Next, we need to set up the container for our action.
If you have never worked with Docker before, this might look a bit confusing.
An advantage of Docker is that other people have done the heavy lifting for us already and provided base images for containers (i.e., execution environments) that we can re-use.
In our case, what we need to do is:

- use a base `node` image (because we want to execute JavaScript),
- copy all files we need into the container,
- install the dependencies, and
- [run the action](https://youtu.be/kYtGl1dX5qI?t=14)

```Dockerfile
FROM node:slim

# The * after package is there also to copy the
# package-lock.json that should is created when
# you run `npm install.`
COPY package*.json ./

RUN npm ci

COPY action.js /action.js

ENTRYPOINT ["node", "/action.js"]
```

We have defined the dependencies and made sure that an environment can run our action.
Since we want to set a different PR status based on the workflow, the action needs two [inputs](https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#inputs).

1. The kind of status we want to set (`pending`, `success`, or `failure`)
2. A description that gives our developers some context

```js
const { Toolkit } = require("actions-toolkit")

const tools = new Toolkit()

// You could also hard-code these as this
// action is bound to one repo but this makes
// copy-pasting this code easier
const { owner, repo } = tools.context.repo

// The SHA of the commit that triggered this action.
// Makes sure this status is associated with the
// correct commit.
const { sha } = tools.context

// These are inputs that we define. You can extend those
// or change their names if you like
const { state, description } = tools.inputs

tools.github.repos
  .createStatus({
    owner,
    repo,
    sha,
    state,
    description,
    target_url: `https://www.github.com/${owner}/${repo}/commit/${sha}/checks`,
  })
  .then(() => {
    tools.exit.success()
  })
  .catch((error) => {
    tools.log.fatal(error)
    tools.exit.failure()
  })
```

You might have noticed that we don't need to do any authentication with GitHub.
The `actions-toolkit` library takes care of this for us as long as it finds a `GITHUB_TOKEN` environment variable with a valid token—more on this in the next section.

## Creating the workflow

The primary goal of our workflow is to run our E2E tests.
If you have skipped the section about adding a custom action, you can skip ahead to [Pointing the test runner at the deployment](#pointing-the-test-runner-at-the-deployment)

But it must also make sure that the correct states are shown in the PR.
Here's what we want to achieve.

- Show a _pending_ state while the deployment isn't ready or the tests are still running
- Show a _success_ state when the tests have finished without errors
- Show a _failure_ state when the tests failed

Let's start with setting the state to `pending`.

### Registering a new status check

We have defined our workflow to run whenever there is a `deployment_status` event.
However, we solely want to set the PR status to pending as long as the deployment is also pending.
Good thing [jobs](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobs) inside a workflow can be [conditional](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idif).

```yaml
jobs:
  set_pending:
    name: Register pending E2E tests state
    # This is the place where we define this job to only
    # run when the deployment state is still "pending".
    if: github.event.deployment_status.state == 'pending'
    runs-on: ubuntu-latest

    steps:
      # This checks out the code of this repository.
      # We need this because this is where our action
      # lives.
      - uses: actions/checkout@v1

      - name: Set status to "pending."
        uses: ./.github/actions/set-pr-status
        env:
          # You don't need to configure any secrets for this
          # to work. GitHub injects the GITHUB_TOKEN automatically.
          # We need it in this step so that our action
          # can talk to the GitHub API.
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          # This is where we define the inputs
          # for this action.
          state: pending
          description: Waiting for E2E results
```

If we now open up a PR, we'll see a new status that reports on the status of our E2E tests.

### Pointing the test runner at the deployment

We're using `cypress` to run E2E tests.
In this scenario, we're using the [`baseUrl` configuration](https://docs.cypress.io/guides/references/configuration.html#Global) to point `cypress` to the location of the Vercel deployment.
We also make sure that this job only runs when the deployment was successful.

```yaml
jobs:
  run_e2e_tests:
    name: Run E2E tests
    # This statement makes sure that this job is only
    # executed when the deployment to Vercel was
    # successful
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    # Thank you cypress for providing a container
    # that works out-of-the-box
    container: cypress/browsers:node11.13.0-chrome73
    env:
      TERM: xterm

    steps:
      - uses: actions/checkout@v1

      - name: Run E2E tests
        run: cypress --config baseUrl=${{ github.event.deployment_status.target_url }}
```

### Reporting the result of the tests back

The last thing we need to do is to set the status to `success` when the tests passed and to `failure` when they did not pass.
We can use the exit code of `cypress` for that.
When a test does not succeed, `cypress` will exit with a non-zero [exit code](https://en.wikipedia.org/wiki/Exit_status) and mark the job as a failure.

```yaml
jobs:
  run_e2e_tests:
  # See above for the complete definition of this job

  steps:
    - name: Set status to "success"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was successful
      if: success()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: success
        description: All tests passed

    - name: Set status to "failure"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was *not* successful
      if: failure()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: failure
        description: Some tests failed
```

That's it! We have created a custom action that we can use to set a status check on a PR and a workflow that will run our end-to-end tests when a deployment is ready.

## Caveats

I like to mention one thing that probably cost me an hour right at the start.
Because I was thinking in the context of a PR I always looked for the deployment action in the "Checks" tab of a single PR.
But that is not how this works.
Since deployments are not necessarily coupled to a PR, GitHub lists them in the "Actions" tab for the whole repository.
I hope this piece of information saves you some time!