Cross Browser Testing with Cypress in CI/CD using Docker
Tags: cypress, docker, cicd, testing
Published: 2023-06-08 14:18:10

_Greetings to all Cypress enthusiasts!_

In this article, we’ll explore one interesting way of running Cypress tests across multiple browsers in CI/CD using Docker. The cool thing here is that with the help of Docker Compose we will set up the simultaneous launch of several Docker containers in which Cypress tests will be executed in parallel in different browsers. The entire workflow will run automatically on the GitHub Actions platform, and as a result, we will get artifacts from running tests in each of the browsers. Get ready to read, it’s going to be very exciting!

## Why Cross Browser Testing?
**Cross Browser compatibility** is one of the most important characteristics of a web application, which implies its equally correct display and functionality in various browsers and their versions.

The modern variety of browsers is determined, among other things, by **_differences in_** the ways of **_rendering_** web application content, when different browser engines (_Blink, WebKit, Gecko, EdgeHTML_) perceive and process HTML tags and CSS styles differently, which naturally affects the appearance and application behavior.

In this regard, the importance of **cross-browser testing** is obvious and extremely clear. Its purpose is to ensure that when a user opens a web application in various browsers and their versions, the correct display of its content is ensured, the integrity of its structure is preserved, there are no functional errors and inconsistencies in performance, layout collapse, overlapping elements on top of each other, etc.

## Cypress Features in Cross Browser Testing
If you are not yet familiar with **[Cypress](https://www.cypress.io/)**:

> **Cypress** _is a JavaScript-based end-to-end testing tool designed for modern web test automation. Cypress allows conducting both full-fledged end-to-end testing with passing user scenarios on a real product, as well as integration testing of individual front-end components. Cypress has emerged as a popular end-to-end testing tool for web applications due to a bunch of its powerful features, user-friendly interface, fast test execution time, easy installation and debugging, etc._

To my mind, Cypress is a real game changer in end-to-end and component testing and it grows at a rapid pace. Among the many benefits of Cypress, I would like to emphasize the high quality of its [documentation](https://docs.cypress.io/guides/overview/why-cypress), as well as the master classes that the Cypress development team conducts and publishes in the public domain, and also a very friendly and responsive [community](https://discord.gg/cypress). Honestly, and as you may have noted from my previous articles I’m a big fan of that wonderful tool!

**_Cypress has the capability to run tests across multiple browsers_**. According to official [documentation](https://docs.cypress.io/guides/guides/cross-browser-testing), Cypress has currently support for [Chrome-family browsers](https://docs.cypress.io/guides/guides/launching-browsers#Chrome-Browsers) (including _Electron_ and Chromium-based _Microsoft Edge_), [WebKit](https://docs.cypress.io/guides/guides/launching-browsers#WebKit-Experimental) (Safari’s browser engine), and _Firefox_. Excluding _[Electron](https://docs.cypress.io/guides/guides/launching-browsers#Electron-Browser)_, any browser you want to run Cypress tests in needs to be installed on your local system or CI environment.

Obviously, it is often not necessary to run all available test suites in different browsers given the increase in test execution time and the associated cost of the required CI infrastructure. Therefore, **_Cypress provides different continuous integration strategies_** for deploying cross-browser testing in CI pipelines depending on the needs of a particular project.

## Benefits of Using Cypress to Optimize Cross Browser Testing in CI
In order to balance costs and available CI resources, as well as the optimal level of confidence in testing, **_Cypress provides the following options for effectively organizing cross-browser testing_**:

1. **_Selecting a specific test suite for a given browser_**. For example, sometimes it makes sense to run all available tests within _Chrome_, but within _Firefox _execute only the happy or critical path related test files, or a directory of specific _“smoke”_ test files using `--spec` flag. In some cases, the priority areas for implementing cross-browser testing may be critical application features or workflows, as well as the most likely user scenarios.

2. **_Flexible setting of the periodic frequency of launching individual browsers_**. For example, running tests within _Chrome_ can be triggered by events in the repository, while within _Firefox_, tests will run on a set schedule based on release periodic frequency. Modern CI pipelines allow you to set the required time and periodic frequency for running workflows.

3. **_Parallel execution of test files for each group_**, where the groups are based on the browsers being tested. For example, using **[Cypress Cloud](https://cloud.cypress.io/login)** allows you to run each browser at different levels of _parallelization_, differentiating the amount of allocated CI resources between browsers depending on the importance of each browser in the testing strategy. For example, tests within _Chrome_ can be run conditionally in parallel on up to four machines, while within _Firefox_ on two, minimizing CI costs.

4. **_Possibility to configure the launching or exclusion of browsers for a specific test/test suite_**. Sometimes it makes sense to run or ignore one or more tests in certain browsers to shorten the duration of a test running. To do this, Cypress allows to specify directly in the configuration of a test or test suite a specific browser to run or exclude, for example: `{ browser: 'firefox' }` or `{ browser: '!chrome' }`.

5. **_Determination of the software deployment environment_**. In cases where the project shows consistently stable behavior in different browsers, it is advisable to set up cross-browser testing only before deploying changes to the _production_ environment.

A thoughtful combination of these Cypress advantages will help you build an optimal cross-browser testing strategy based on the needs of a particular project.

## Why Use Docker in Cross Browser Testing?
There are various approaches to implementing cross-browser testing with Cypress, one of which is to set up the automatic running of Cypress tests in chosen CI platform using **[Docker](https://www.docker.com/)**.

Docker makes it much easier to set up and maintain a cross-browser test environment. By encapsulating the entire test stack, including different browsers in isolated Docker containers, it is possible to reproduce a stable and consistent environment for running tests in different browsers, regardless of the servers running the containers.

Using **[Docker Compose](https://docs.docker.com/compose/)** allows us to effortlessly scale our testing infrastructure by distributing our workload across multiple containers. So, it is possible to define several containers for different browsers and run them simultaneously using one command, or rather one configuration file. Parallel execution of tests in different browsers obviously significantly speeds up the overall testing process.

Using the **official Cypress Docker [images](https://github.com/cypress-io/cypress-docker-images)** with pre-installed browsers as a base layer eliminates the need to install browsers on servers running Docker containers. You can read more about the benefits of using Docker in testing in my previous [article](https://medium.com/testing-with-cypress/running-cypress-tests-in-docker-containers-using-different-docker-images-2dee3450881e) about running Cypress tests in Docker containers.

## Moving from theory to practice…
In this article, I use **[GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions)** as my continuous integration platform. It should be noted that _**Cypress provides many useful [examples](https://github.com/cypress-io/github-action) of configuration files for setting up GitHub Actions workflows**_, based on which it is quite simple to organize various options for running tests in several browsers. For example, you can create a workflow for each browser separately and activate the created workflows depending on the specified trigger events in the repository. Obviously, this is very convenient.

This article provides a simplified example of creating multiple containers to run tests within different browsers simultaneously using Docker Compose.

The idea is pretty simple — let’s say we need to run a specific Cypress test suite within four browsers — _Google Chrome, Firefox, Microsoft Edge_, and _Electron_. Trigger events for automaticall ystarting a workflow in the repository should be a _push_ event to the _main_ branch, an _open_ or _reopened pull request_, and a _scheduled_ start, for example, every Friday at 2 am.

Also, during the execution of the workflow, it is necessary to obtain **artifacts** with the results of running tests in each of the browsers — videos and screenshots in case any tests fail.

One possible solution to this task is to create a custom Docker image based on one of the **official Cypress Docker [images](https://github.com/cypress-io/cypress-docker-images)**, build and simultaneously run four containers from the created image, in each of which our Cypress tests will be executed in a specific browser.

## Briefly about the test project
To demonstrate how to run Cypress tests across multiple browsers, I use the very simple _Cypress-Docker_ project to test my blog **_[“Testing with Cypress”](https://medium.com/testing-with-cypress)_** on Medium. The project already has some dependencies installed — **_[Cypress 12.13.0](https://www.npmjs.com/package/cypress/v/12.13.0)_** and **_[Typescript 5.0.4](https://www.npmjs.com/package/typescript/v/5.0.4)_**, and also has _spec.cy.ts_ file with a set of three trivial tests for the blog’s homepage:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sjojx5f7mz52gf5sar7b.png)

By running it locally within _Chrome_ browser, we make sure that all tests pass successfully:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oao2338a0f34ttb4kllp.png)

Obviously, these tests are only demonstration examples and do not indicate the correct display and functionality of the website in different browsers.

To run tests on the GitHub Actions platform, this project is hosted on GitHub.

## Building a Docker image
As a base layer for building the image, the official Docker image **[Cypress/Browsers](https://hub.docker.com/r/cypress/browsers)** was taken, which includes all operating system dependencies and some browsers.

The most current version of the image at the moment — **[node-18.16.0-chrome-113.0.5672.92–1-ff-113.0-edge-113.0.1774.35–1](https://hub.docker.com/layers/cypress/browsers/node-18.16.0-chrome-113.0.5672.92-1-ff-113.0-edge-113.0.1774.35-1/images/sha256-478b68c50a2544d0b8f00e3b79a09e084e16a8a1637096404c27c319d185b41d?context=explore)** includes pre-installed _Node.js 18.16.0_, as well as three browsers — _Google Chrome, Firefox_ and _Microsoft Edge_. Given that the _Electron_ browser is pre-installed in Cypress, we will have all the necessary browsers at our disposal to carry out cross-browser test execution in accordance with the initial task.

The final **Dockerfile** for building the required image will look like this:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/plta3yq6xht4nngdz7ne.png)

Firstly, `FROM` instruction defines a base image, all of whose dependencies and configurations will be included in the generated image.

The next `WORKDIR` step creates the `/e2e` working directory, in which all subsequent commands will be executed.

Next, in `COPY `instruction the _package.json_ and _cypress.config.ts_ files, as well as the _cypress_ folder, including the _spec_ file, will be copied from the repository to the working directory inside the image.

Then `RUN` instruction specifies two commands to run, in particular, `npm i` — to install the necessary dependencies in the working directory of the image and `npx cypress info` — to display information about Cypress, current browsers detected by Cypress, and so on.

The last step in `ENTRYPOINT` instruction the command in _exec_ form is defined to run Cypress in headless mode in containers generated from this image.

## Setting up running containers with Docker Compose
As mentioned earlier, after building the Docker image, _four_ containers will be created based on it. In each container, Cypress tests will be run in a specific browser. To simultaneously run containers with a single command, it is advisable to use a tool such as **[Docker Compose](https://docs.docker.com/compose/)**. To describe the process of loading and configuring containers, the YAML configuration file should include the following:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5pndq7wm5cpyhw88eyu2.png)

As we can see, in the configuration file we define _four_ `services` (containers) — `e2e-chrome`, `e2e-firefox`, `e2e-edge` and `e2e-electron`, each of which uses the Docker image created based on the _Dockerfile_ located in the same directory (`build` keys).

Next, the values ​​of the `command` keys set the commands to launch Cypress in browsers with the given names. It is worth noting here that in the `e2e-firefox` service, the command was supplemented by a configuration change due to an open [issue](https://github.com/cypress-io/cypress/issues/18415) with recording video when using the _Firefox_ browser in Cypress.

The next step is to mount volumes (`volumes` keys) for each service and set the appropriate mappings to gain access to artifacts outside of containers. In essence, this means that the videos and screenshots generated during the launch of Cypress and written into containers will actually be stored in the GitHub Actions virtual environment at the specified paths relative to the workspace of the launched workflow. This will allow, in case any tests fail, to extract artifacts from the `./artifacts` directory during the execution of the GitHub Actions workflow, which is required according to the initial task.

## Setting up the Workflow in GitHub Actions
To run the workflow, we’re going to use following workflow _e2e.yml_ file which is located in _.github/workflows_ directory of the project:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ecpf6xjwnj14dtvzm6m5.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/acvi4k71hjy2k7jw69jp.png)

Let’s analyze it in more detail. By using `on` key we define trigger events that automatically activate the workflow according to the initial task conditions:

`push` — when pushing changes to the _main_ branch
`pull_request` — when _opening_ or _reopening a pull request_
`schedule` — run a _scheduled_ workflow every Friday at 2am

Next, we define one job in the workflow, `cypress-run`. The job will run in a virtual machine with the latest version of the _Ubuntu Linux_ operating system within the configured `timeout` of _5 minutes_ to ensure that an accidental hang does not use up extra CI minutes.

The `steps` key combines all the necessary steps to complete the task. First, custom application `actions/checkout@v3` is launched, which extracts the repository into the virtual machine, sequentially performing the necessary actions, including checking the _git_ version, creating the necessary folders, authorizing, etc.

Next, we see the main step of the job — `Run docker-compose`, which executes `docker-compose up` command to build and run four Docker containers with the specified names (`e2e-chrome`, `e2e-firefox`, `e2e-edge` and `e2e-electron`) defined in Docker Compose configurations. In this case `-d` flag means running containers in the background to allow the workflow to continue executing subsequent steps.

The purpose of the next step is to provide visibility into the logs of running Docker containers in order to monitor them and control the process of starting services before moving on to subsequent workflow steps. The `docker-compose logs -f` command displays logs of all running containers in real-time. At this step, it is possible to track the execution of Cypress tests step by step in each of the _four_ browsers.

The next four steps are identical in nature and ensure that the workflow artifacts from each container are downloaded for access after the workflow completes. The custom application `actions/upload-artifact@v3` takes the paths provided as input and uploads folders with generated by Cypress videos and screenshots if the test fails. This will make the artifacts from each container available in the workflow summary. In addition, the behavior of the action in case no artifact files are found is also configured here, as well as the storage period for artifacts (_5 days_).

## Starting the Workflow
To activate the workflow, let’s create a _trigger event_. For this let’s make a small change to one of the existing tests, for example, add “.” at the end of the expected header text (highlighted) for the test to fail:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4auxkkqiwehw267lb8dn.png)

Next, let’s commit and push the change to the _main_ branch:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/21s01o52sogtsxrl2bs2.png)

Voila, the workflow is started! Let’s go to the project repository on GitHub and open the summary of the last workflow in the _Actions_ tab:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8zqtyhnr8kanq6hfi1wk.png)

In the log of the completed `cypress-run` job, we make sure that all steps were completed successfully:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uywwcn04360wxo9rhln9.png)

In particular, at the `Run docker-compose` step, a Docker image was built based on the previously described _Dockerfile_, from which four containers were generated. During the build of the image, `npx cypress info` command was executed and information about detected browsers was displayed in the log, as well as other characteristics of the test environment — the operating system, versions of Node.js, Cypress, etc.:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lk8usizsreljgfva8c7k.png)

At the next step, processes were simultaneously launched in the created containers:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fa7fvuy5p7b9s0rt7ddh.png)

to be more precise, parallel execution of Cypress tests was launched in four browsers:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vhp0fv71974vg3tpwr1n.png)

We can see logs from each container about the progress of the test execution:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bxessj8icvov5ofm004w.png)

As a result, the first test failed as expected, while the next two passed successfully in each of the _four_ browsers:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f447gxdjsgq8kpyltfgz.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eg56smkkn3ybogo4zpmb.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/87ad8h2cheynmiomfdy2.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tbwg1fwxux96mp4yhxhp.png)

## Validate Workflow Artifacts
As you may have noticed earlier, the workflow summary contains information about the downloaded artifacts:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dmbtfm7c0c9c8uzylky8.png)

After downloading the artifacts from the workflow summary in ZIP format, we verify that we have videos and screenshots of the failing test in each browser (except the video in _Firefox_):

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mwf1lxpvfqpcyxj5pj03.png)

Let’s check that in case of successful completion of all tests, the workflow artifacts will not be loaded. To do this, let’s fix the first test, make a commit, and push the change to the _main_ branch again:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8vm3nylk5e9di4de3oka.png)

After the workflow is completed, we make sure that artifacts were not loaded from any container due to their absence:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ol3mocdcz1flhha96soz.png)

## Final thoughts
In conclusion, it should be noted that the optimization of cross-browser testing with Cypress based on the approach described in this article has a bunch of obvious advantages.

In particular, the **_simultaneous launch and parallel execution of tests in several browsers_** are guaranteed. Containerization ensures that the **_test environment is consistent and reproducible_** when running Cypress tests on different browsers.

Moreover, it is **_easier to set up and maintain a test environment_** based on a single configuration file. The **_test infrastructure scales easily_** depending on the desired level of parallelism for each run or the number of browsers being tested, etc.

In general, all this allows you to **_increase the efficiency of using available CI resources, reduce the total time for executing tests, expand test coverage in several browsers, providing the optimal level of reliability, taking into account the specifics of a particular project_**.

That’s about it. If you found this useful, share it with a friend or community. Maybe there’s someone who will benefit from it as well. To continue your journey with me and get more information about testing with the awesome **[Cypress](https://www.cypress.io/)** tool, you might be interested in subscribing to my blog **[“Testing with Cypress”](https://medium.com/testing-with-cypress)** and get notified when there’s a new useful article.

The source code of all examples presented in this article can be found in the [repository](https://github.com/Sanzhanov/Blog-Testing-with-Cypress) of the blog on GitHub.

_Thank you for your attention! Happy testing!_