Pytest vs Cypress: A fair fight in UI testing?
Tags: pytest, selenium, python
Published: 2020-06-17 04:22:59

---
title: Pytest vs Cypress: A fair fight in UI testing?
published: true
description: A Battle of End-to-End Testing Frameworks
tags: #pytest, #selenium, #python
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/knyj4c1i7qoo37nfpqw8.png
---

Right now, we're witnessing the greatest disruption of the test automation industry that we've ever seen. New tools being developed, and existing tools are getting significant improvements. Today, let's talk about two major test frameworks that have lots of users according to GitHub:

![pytest vs cypress on GitHub](https://dev-to-uploads.s3.amazonaws.com/i/xybp1thyezzz8q5tk47j.png)

Some of you may already be familiar with **cypress**: a known framework for end-to-end testing. Unlike cypress, **pytest** is mostly known as just being a unit-testing framework (for Python). Less known about pytest is that it has a major plugin ecosystem for expanding its abilities. Just like Batman has a tool belt for letting him do things that he can't do with his bare hands, **pytest** has plugins for allowing it to do much more, such as browser testing, multi-threading tests, and creating test reports. Before we can go into the **pytest** vs **cypress** comparison, let's talk about an essential component needed for UI testing with pytest, which is the **selenium** browser automation library.

In the past, you've probably heard of comparisons being done between **cypress** and **selenium**. This raw comparison might not have been fair because Selenium clearly states on their website: "Selenium automates browsers. That's it! What you do with that power is entirely up to you." (source: [selenium.dev](https://www.selenium.dev/)) Nowhere on their website did Selenium say anything such as them being "a complete test framework", etc. They specifically wanted to give developers and test automation engineers the ability to create their own test frameworks that use **selenium**. A more fair comparison would be cypress vs a test framework that <i>uses</i> selenium.

Fast-forward to today: Lots of companies want to get a lot of testing done very quickly, but they don't have the time they need to build a complete UI testing framework for their testing. This has led to a lot of corner-cutting (aka sloppy code), and false blame against perfectly solid automation libraries that work great when used properly. One thing to keep in mind when using raw Selenium methods is that if you try interacting with page elements that haven't fully loaded (or elements that haven't been re-found after a page reload), then you may see errors like this:
* ``NoSuchElementException``
* ``ElementNotVisibleException``
* ``MoveTargetOutOfBoundsException``
* ``StaleElementReferenceException``
* ``ElementNotInteractableException``

These errors are <i>completely</i> preventable if your test framework wraps **selenium** methods with the appropriate code that smart-waits for page elements to be fully interactable before interacting with them (or timeout/fail the test if the element still doesn't appear after a set number of seconds). The good news is that a test framework with selenium smart-waiting already exists as a **pytest** plugin: [SeleniumBase](https://github.com/seleniumbase/SeleniumBase)!

<img src="https://dev-to-uploads.s3.amazonaws.com/i/7e6vmjhnbi9te1x2ne36.png" alt="SeleniumBase" width="260" />

**SeleniumBase** bridges the gap between **pytest** and **selenium** for UI testing. With pytest, you have a unit-testing framework without a built-in web automation component. With selenium, you have a web automation library without a built-in test framework. Both sides of the bridge have long been very popular and very powerful, but now there's a special bridge that seamlessly brings the two sides together in one very powerful, but simple, package. And this package, **SeleniumBase**, exists as a pytest plugin, which uses the selenium library effectively.

Here's an example **SeleniumBase** test:
```python
from seleniumbase import BaseCase

class MyTestClass(BaseCase):
    def test_basic(self):
        self.open("https://store.xkcd.com/search")
        self.type('input[name="q"]', "xkcd book\n")
        self.assert_text("xkcd: volume 0", "h3")
        self.open("https://xkcd.com/353/")
        self.assert_title("xkcd: Python")
        self.assert_element('img[alt="Python"]')
        self.click('a[rel="license"]')
        self.assert_text("free to copy and reuse")
        self.go_back()
        self.click_link_text("About")
        self.assert_exact_text("xkcd.com", "h2")
```
That test is from the latest version of [SeleniumBase/examples/my_first_test.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/my_first_test.py), which you can run using pytest:

```bash
pytest my_first_test.py
```

As you can see, it's very easy to open a URL, type text, click, and assert various elements on a page. For a more advanced example (with hovering dropdowns, iFrames, select options, sliders, finding broken links, and detecting JavaScript errors), see [SeleniumBase/examples/test_demo_site.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/test_demo_site.py)

Now let's compare that with a **Cypress** example:
```javascript
describe('My First Test', () => {
  it('Gets, types and asserts', () => {
    cy.visit('https://example.cypress.io')
    cy.contains('type').click()
    cy.url().should('include', '/commands/actions')
    cy.get('.action-email')
      .type('fake@email.com')
      .should('have.value', 'fake@email.com')
  })
})
```

Aside from the obvious differences that Cypress tests are written in JavaScript, and SeleniumBase tests are written in Python, you'll see that both examples are written using simple code, both examples are easy to run, and both examples run reliably.

For test reports:
* **cypress** has the [Cypress Dashboard Service](https://www.cypress.io/dashboard).
* **pytest** has the **pytest-html** plugin, which **SeleniumBase** modifies to include screenshots of failing tests at the moment they failed:

![SeleniumBase pytest-html report](https://dev-to-uploads.s3.amazonaws.com/i/h1misazs70sefqvg03de.png)

SeleniumBase doesn't have its own dashboard service, but you can easily run your SeleniumBase tests from any cloud service you desire. For open-source projects, I've personally used three free tools: First Travis-CI, then Azure Pipelines, and finally GitHub Actions, which allows you to keep all your work and test results in GitHub. Here's an example of SeleniumBase results from GitHub Actions:

![GitHub Actions results for SeleniumBase](https://dev-to-uploads.s3.amazonaws.com/i/daransz31i2udfmpj7ni.png)

When using SeleniumBase to test applications that aren't open source, I've primarily used Jenkins. If you're doing test automation on a tight budget for closed-source projects, the most cost-effective Jenkins solution I've found so far is [Google Cloud's $13.61 USD/month Bitnami Jenkins instance](https://console.cloud.google.com/marketplace/details/bitnami-launchpad/jenkins), which supports unlimited users and unlimited test runs (limited only by its CPU, which is enough to handle five simultaneous browser tests without any noticeable lag). As for Cypress tests, those can also be run with GitHub Actions and on Jenkins.

Now that we've covered the basics, lets move onto limitations when it comes to Cypress vs Pytest. Cypress actually has a page on its website dedicated to its limitations, aka "trade-offs": [docs.cypress.io -> trade-offs.html](https://docs.cypress.io/guides/references/trade-offs.html).

These **Cypress limitations** include:
* Limited iFrame support.
* No multiple tabs.
* No multiple browsers at the same time.
* Each test is bound to a single origin domain.
* Missing mobile support.
* Not a general purpose automation tool.

With **pytest and SeleniumBase**, all of the following features exist:
* Full iFrame support.
* Open as many tabs as you want in tests.
* Open as many browsers as you want in tests.
* No limitations on multi-origin domain navigation.
* Uses Chrome's mobile-device emulator for mobile tests.
* Can be used for general purpose automation.

The only limitation with **SeleniumBase** is that you must use **Python**! (And for Python users that's not a limitation... that's a feature!) The good news is that: **You can also make JavaScript calls with SeleniumBase!**
This is done with:
```python
self.execute_script(JAVASCRIPT)
```
This ability to execute JavaScript inside SeleniumBase Python scripts has led to some powerful new features, such as the website tour builder that's part of SeleniumBase ([learn more about the SeleniumBase tour-builder here](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/tour_examples/ReadMe.md)):

![SeleniumBase website tour builder](https://dev-to-uploads.s3.amazonaws.com/i/pgfkmahizmgnbwck6skg.gif)

As for SeleniumBase's mobile mode, just add:
``--mobile`` to the pytest run command of any SeleniumBase test to run it using Chrome's mobile device emulator:

![SeleniumBase mobile test](https://dev-to-uploads.s3.amazonaws.com/i/szm359eleer7syad8lc2.gif)

The fastest way to get started with SeleniumBase is by cloning it from GitHub so that you can run all the included example tests, but you can also install seleniumbase from [PyPI](https://pypi.org/project/seleniumbase/) with pip, then do:
``seleniumbase mkdir TEST_FOLDER`` to create a folder anywhere that includes example tests for you to run.

At the time this article was published, the latest version of seleniumbase was "v1.40.2", so if you're using an older version of SeleniumBase, some of the features mentioned here might not exist until you upgrade.

Here are some links to material discussed in this article:
* [Cypress on GitHub](https://github.com/cypress-io/cypress)
* [Cypress Docs Site](https://www.cypress.io/)
* [SeleniumBase on GitHub](https://github.com/seleniumbase/SeleniumBase)
* [SeleniumBase Docs Site](https://seleniumbase.io/)

Now that you've had a chance to look at both Cypress and Pytest (with the SeleniumBase plugin), you must decide for yourself where to go from here. I'm biased of course (being a Python developer). I started learning Python in 2008 at a company called ITA Software, which was later acquired by Google. And I started learning Selenium in 2011 at a company called HubSpot, which funded the creation of the test framework that I would later rename to **SeleniumBase** in 2014 after the project got approval to be open-sourced. (Trivia: I decided on the name "SeleniumBase" on the day Google acquired Firebase. I heard about that acquisition on the news, and I liked the sound of "Firebase", so I took that idea and added "Base" onto "Selenium", and that became the name of the framework ever since.)

I hope that you got a few key takeaways from this article: Mainly, that when used correctly, **Selenium** is awesome for UI testing. When paired together with **pytest** (and a good "bridge" like **SeleniumBase**), **Selenium** is even more awesome. People have spent a lot of time <i>unjustly</i> trying to replace Selenium, when in reality, all they needed to do was replace their implementation strategy.

Thank you to both the **pytest** and **Selenium** teams for all the hard work you've done in making great testing tools. Here's a shout-out to some of the major contributors of these projects: [Simon Stewart](https://github.com/shs96c), [Bruno Oliveira](https://github.com/nicoddemus), [Jim Evans](https://github.com/jimevans), [Ronny Pfannschmidt](https://github.com/RonnyPfannschmidt), [David Burns](https://github.com/AutomatedTester), [Florian Bruhin](https://github.com/The-Compiler), [Dave Hunt](https://github.com/davehunt), [Jason Huggins](https://github.com/hugs), [Diego Molina](https://github.com/diemol), and many more key contributors that I haven't listed here. There will always be competition in the test automation space, but together, we can continue to create a legacy for test automation engineers of today and the future.