Using Vercel's instant rollback feature in your own CI/CD pipeline
Tags: vercel, nextjs, cicd, devops
Published: 2023-12-02 17:24:09

Vercel, a Platform as a Service (PaaS) for hosting Next.js web applications, offers some really convenient features like instant rollbacks. This post explores how to use this feature in a custom CI pipeline to allow colleagues without a Vercel login to carry out rollbacks.

Vercel is built on a serverless architecture, so it's cheap for them to keep copies of all your previous deployments. This allows for some cool features, like a [`vercel bisect`](https://vercel.com/docs/cli/bisect) CLI command that works like [`git bisect`](https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination), except instead of checking out file changes locally to find a bug, it gives you URLs of each deployment to test before telling you which deployment introduced an issue. 

## Vercel instant rollbacks

Another cool feature is [instant rollbacks](https://vercel.com/docs/deployments/instant-rollback). If you deploy a change that causes issues, rather than doing a `git revert` or a hotfix and then re-running your build/test/deploy pipeline you can instead instantly revert to the last deployment, saving lots of time and stress. You can do a rollback via logging into Vercel's admin UI, via their CLI, or via an undocumented REST endpoint.

A downside to Vercel is that it's a premium service with a premium cost for the higher-end tiers. You're charged per-seat (per developer), so in many organisations not everyone will have a Vercel login. This is a bit of a problem for instant rollbacks – what if the developer who spots the production issue doesn't have a Vercel account so can't use their admin UI, CLI, or REST API?

To work around this issue at work I recently added a 'Rollback' step to the end of our CI pipeline. Every developer has access to the pipeline so will be able to initiate a rollback via GitLab CI instead of Vercel. The pipeline is authorised to connect to Vercel's API using an API token associated with an existing Vercel account.

## How I did it

Firstly I generated a [Vercel API token](https://vercel.com/guides/how-do-i-use-a-vercel-api-access-token) and added it to GitLab CI as an environment variable called `VERCEL_TOKEN`. I also created a `VERCEL_TEAM_ID` and `VERCEL_PROJECT_ID` using the team and project IDs from the Vercel web interface.

[Vercel's REST API documentation](https://vercel.com/docs/rest-api/endpoints) is missing docs for a 'rollback' endpoint, but Vercel's CLI is hosted on GitHub and you can [see how they are using their own REST API (a good sign!) for the `rollback` CLI command](https://github.com/vercel/vercel/blob/a73ec6343f9b80f9149272724da750c851cb473a/packages/cli/src/commands/rollback/request-rollback.ts#L33-L36):

```javascript
  await client.fetch(`/v9/projects/${project.id}/rollback/${deployment.id}`, {
    body: {}, // required
    method: 'POST',
  });
```

## The rollback process, via API

Before we can send a rollback request to the API we need to find the previous production release's deployment ID. Here's how I did that in Bash, using [jq](https://jqlang.github.io/jq/):

```bash
# Fetch the rollback deployment ID
ROLLBACK_DEPLOYMENT_ID=$(curl -s "https://api.vercel.com/v6/deployments?teamId=$VERCEL_TEAM_ID&projectId=$VERCEL_PROJECT_ID&limit=2&rollbackCandidate=true&state=READY&target=production" -H "Authorization: Bearer $VERCEL_TOKEN" | jq -r '.deployments[1].uid')
echo "Rolling back to the previous deployment ID: $ROLLBACK_DEPLOYMENT_ID"
```

Note the query string parameters in that request:
* `limit=2`, as we want to get the previous production deployment, not the current one
* `rollbackCandidate=true`, as we need a deployment we can rollback to
* `state=READY`, to filter out deployments that might be in-progress
* `target=production`, because we are not interested in preview deployments

We pipe (`|`) the silent `curl` output to jq and use it to get the second `uid` from the results. We now have a variable called `ROLLBACK_DEPLOYMENT_ID` that we can include in a `POST` request to Vercel's undocumented rollback endpoint:

```bash
# Initiate a rollback to the previous deployment
curl -s -X POST "https://api.vercel.com/v9/projects/${VERCEL_PROJECT_ID}/rollback/${ROLLBACK_DEPLOYMENT_ID}?teamId=${VERCEL_TEAM_ID}" -H "Authorization: Bearer $VERCEL_TOKEN" -H "Content-Type: application/json"
```

If both those commands are successful then your site should rollback to the previous production release, hopefully before too many of your users have noticed the issue.

## GitLab CI example

I added the above commands to a `scripts/rollback-current-prod-deployment.sh` file, made it executable, and then referenced it in a GitLab CI job like this:

```yaml
Rollback current prod deployment:
  image: registry.gitlab.com/gitlab-ci-utils/curl-jq:latest
  stage: deploy
  needs:
    - job: Upload to Vercel prod URL
      optional: true
  script:
    - './scripts/rollback-current-prod-deployment.sh'
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      when: never
    - if: $CI_COMMIT_BRANCH != "main"
      when: never
    - when: manual
```

This job:
* only runs manually
* only runs after the 'Upload to Vercel prod URL' job
* uses a Docker image that contains both `curl` and `jq`. (Note that [despite the name, this image is not an official GitLab product](https://gitlab.com/gitlab-ci-utils#:~:text=Note%20that%20this%20group%20is%20not%20affiliated%20with%20GitLab%20(although%20we%20do%20extensively%20use%20GitLab).))

It'd be a very similar approach to use this with GitHub Actions or any other CI/CD pipeline provider.

## How to take this idea further

I made a few other additions that are application-specific so I haven't covered here.

### An 'undo rollback' job

I created a job to undo the rollback, in case it was triggered accidentally or was no longer needed. This means the original production deployment can be quickly restored, without re-running the whole pipeline

I did this by storing the original deployment ID and writing it to a file that a second pipeline step could read from.

### Confirmation that the rollback worked by displaying the deployed version after the rollback succeeded

I created an `/api/version` endpoint in my application that returned the version of the site in a JSON object. I query this after the rollback job so I can give the pipeline user confirmation that the rollback succeeded. I set the API route to have no caching with a `max-age` header (`public, max-age=0, must-revalidate`), but just to be sure I added some cache busting to the CI job too:

```bash
# Fetch version information twice to help bust cache and stale-while-revalidate
cache_buster=$(date +%s)
curl -s -o /dev/null "https://example.com/api/version?cache_buster=$cache_buster"
VERSION_INFO=$(curl -s "https://example.com/api/version?cache_buster=$cache_buster")
echo "Version after rollback: $(echo $VERSION_INFO | jq -r '.version')"
```