How To Stop Test Suite after N Test Failures in Pytest?
Tags: jesttesting, automationtesting, seleniumtesting, pytest
Published: 2023-02-27 05:51:32

An exhaustive test-suite comprises of many test cases that test different features on various combinations of browsers, platforms, and devices Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing [test automation.](https://www.lambdatest.com/selenium-automation?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=webpage)

The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).

In case you are new to pytest framework checkout this video to know more about pytest framework & its installation.

{%youtube _bm3wIu3xTg%}

In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.

## Using @pytest.mark.incremental Decorator To Stop Test Suite After N Test Failures

The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.

PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.

**conftest.py (to skip test in Python with pytest)**

    import pytest
    from selenium import webdriver
    import urllib3
    import warnings
     
    def pytest_runtest_makereport(item, call):
        if "incremental" in item.keywords:
            if call.excinfo is not None:
                parent = item.parent
                parent._previousfailed = item
     
    def pytest_runtest_setup(item):
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" % previousfailed.name)

The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) in order to skip the test in Python with pytest.

This PyTest Tutorial for beginners and professionals will help you learn how to use PyTest framework with Selenium and Python for performing Selenium automation testing.

{%youtube UzkuOACmBpA%}

**_Leverage the power of LambdaTest to execute your [Jest](https://www.lambdatest.com/jest?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=learning_hub) automation tests at scale, simultaneously on various operating system and browser configurations. Learn more about LambdaTest's capabilities._**

## How To Skip Test In Python With pytest Using @pytest.mark.incremental?

To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an [automated browser testing](https://www.lambdatest.com/?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=webpage) example that contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.

Fixture with scope as the class is used to insatiate the Chrome browser instance. Implementation to skip test in Python with pytest, is also included in conftest.py.

**Conftest.py**

    import pytest
    from selenium import webdriver
    import urllib3
    import warnings
     
    def pytest_runtest_makereport(item, call):
        if "incremental" in item.keywords:
            if call.excinfo is not None:
                parent = item.parent
                parent._previousfailed = item
     
    def pytest_runtest_setup(item):
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" % previousfailed.name)
            
    @pytest.fixture(scope="class")
    def driver_init(request):
        web_driver = webdriver.Chrome()
        request.cls.driver = web_driver
        yield
        web_driver.close()
        # web_driver.quit()

**test_step.py**

    Facebook
    Twitter
    LinkedIn
    How To Stop Test Suite after N Test Failures in Pytest?
    
    Himanshu Sheth
    Posted On: May 26, 2020
    
    view count61133 Views
    
    Read time7 Min Read
    
    Home>Blog>How To Stop Test Suite after N Test Failures in Pytest?
    This article is a part of our Content Hub. For more in-depth resources, check out our content hub on Selenium pytest Tutorial.
    
    An exhaustive test-suite comprises of many test cases that test different features on various combinations of browsers, platforms, and devices Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing test automation.
    
    The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).
    
    In case you are new to pytest framework checkout this video to know more about pytest framework & its installation.
    
    Youtube Thubnail
    In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.
    
    TABLE OF CONTENT
    
    Using @pytest.mark.incremental decorator To Stop Test Suite after N Test Failures
    How To Skip Test In Python With pytest Using @pytest.mark.incremental?
    Using maxfail Command Line Option To Stop Test Suite after N Test Failures
    How To Skip Test In Python With pytest Using maxfail?
    Using @pytest.mark.incremental Decorator To Stop Test Suite After N Test Failures
    The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.
    
    PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.
    
    conftest.py (to skip test in Python with pytest)
    
    import pytest
    from selenium import webdriver
    import urllib3
    import warnings
     
    def pytest_runtest_makereport(item, call):
        if "incremental" in item.keywords:
            if call.excinfo is not None:
                parent = item.parent
                parent._previousfailed = item
     
    def pytest_runtest_setup(item):
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" % previousfailed.name)
    view rawconftest.py hosted with ❤ by GitHub
    The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) in order to skip the test in Python with pytest.
    
    This PyTest Tutorial for beginners and professionals will help you learn how to use PyTest framework with Selenium and Python for performing Selenium automation testing.
    
    
    How To Skip Test In Python With pytest Using @pytest.mark.incremental?
    To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an automated browser testing example that contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.
    
    Fixture with scope as the class is used to insatiate the Chrome browser instance. Implementation to skip test in Python with pytest, is also included in conftest.py.
    
    Conftest.py
    
    import pytest
    from selenium import webdriver
    import urllib3
    import warnings
     
    def pytest_runtest_makereport(item, call):
        if "incremental" in item.keywords:
            if call.excinfo is not None:
                parent = item.parent
                parent._previousfailed = item
     
    def pytest_runtest_setup(item):
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" % previousfailed.name)
            
    @pytest.fixture(scope="class")
    def driver_init(request):
        web_driver = webdriver.Chrome()
        request.cls.driver = web_driver
        yield
        web_driver.close()
        # web_driver.quit()
    view rawConftest.py hosted with ❤ by GitHub
    test_step.py
    
    #Using @pytest.mark.incremental decorator to stop test suite after n test failure using Selenium test automation in Python with pytest 
    import pytest
    import pytest_html
    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.common.keys import Keys
    import time
    from time import sleep
    import sys
    import urllib3
    import warnings
    # @pytest.mark.incremental Stop Test Suite after N Test Failures in Pytest
    @pytest.mark.usefixtures("driver_init")
    @pytest.mark.incremental
    class Test_Scenario_1:
        def test_1(self):
            self.driver.get('https://www.lambdatest.com/blog/')
            self.driver.maximize_window()
     
            expected_title = "LambdaTest | A Cross Browser Testing Blog"
            assert expected_title ==  self.driver.title
            time.sleep(5)
     
        def test_2(self):
            self.driver.get('https://www.google.com/')
            self.driver.maximize_window()
            title = "Google"
            assert title == self.driver.title
     
            search_text = "LambdaTest"
            search_box = self.driver.find_element_by_xpath("//input[@name='q']")
            search_box.send_keys(search_text)
     
            time.sleep(5)
            search_box.submit()
     
            time.sleep(5)
            
            # Click on the LambdaTest HomePage Link
            # This test will fail as the titles will not match
            title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
            lt_link = self.driver.find_element_by_xpath("//h3[.='LambdaTest: Cross Browser Testing Tools | Free Automated ...']")
            lt_link.click()
     
            time.sleep(10)
            assert title == self.driver.title   
            time.sleep(2)
     
        # As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
        # will be marked as xfailed
        def test_3(self):
            self.driver.get('https://www.lambdatest.com/')
            self.driver.maximize_window()
     
            expected_title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest"
            assert expected_title ==  self.driver.title
            time.sleep(5)
     
    @pytest.mark.usefixtures("driver_init")
    @pytest.mark.incremental
    class Test_Scenario_2:
        def test_4(self):
            self.driver.get('https://lambdatest.github.io/sample-todo-app/')
            self.driver.maximize_window()
     
            self.driver.find_element_by_name("li1").click()
            self.driver.find_element_by_name("li2").click()
     
            title = "Sample page - lambdatest.com"
            assert title ==  self.driver.title
     
            sample_text = "Happy Testing at LambdaTest"
            email_text_field =  self.driver.find_element_by_id("sampletodotext")
            email_text_field.send_keys(sample_text)
            time.sleep(5)
     
            self.driver.find_element_by_id("addbutton").click()
            time.sleep(5)
     
            assert self.driver.find_element_by_xpath("//span[.='Happy Testing at LambdaTest']").text == sample_text

As seen in the implementation, incremental testing is applied to class Test_Scenario_2 and Test_Scenario_2 via the @pytest.mark.incremental decorator.

    ......................................
    ......................................
    from time import sleep
    import sys
    import urllib3
    import warnings
    ......................................
    ......................................
     
    @pytest.mark.usefixtures("driver_init")
    @pytest.mark.incremental
    class Test_Scenario_1:
     ..................
     ..................
     def test_2(self):
            self.driver.get('https://www.google.com/')
            self.driver.maximize_window()
            title = "Google"
            assert title == self.driver.title
     
            search_text = "LambdaTest"
            search_box = self.driver.find_element_by_xpath("//input[@name='q']")
            search_box.send_keys(search_text)
     
            time.sleep(5)
            search_box.submit()
     
            time.sleep(5)
            
            # Click on the LambdaTest HomePage Link
            # This test will fail as the titles will not match
            title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
     ..................
     ..................
     
    @pytest.mark.usefixtures("driver_init")
    @pytest.mark.incremental
    class Test_Scenario_2:
     ..................
     ..................

In test case test_2(), Google search for LambdaTest is performed. Once the search results are ready, a click on the first search result is done and the page title is compared with the expected title. For testing, we have deliberately introduced an error in the code and the Selenium test automation case fails as the titles do not match.

    class Test_Scenario_1:
     ..................
     ..................
     def test_2(self):
            self.driver.get('https://www.google.com/')
            .................
       ..................
            search_text = "LambdaTest"
            search_box = self.driver.find_element_by_xpath("//input[@name='q']")
            search_box.send_keys(search_text)
     
            time.sleep(5)
            search_box.submit()
     
            time.sleep(5)
            
            # Click on the LambdaTest HomePage Link
            # This test will fail as the titles will not match
    title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
     ..................
     ..................
     
     # As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
     # will be marked as xfailed
          def test_3(self):
       ..................
       ..................
    @pytest.mark.usefixtures("driver_init")
    @pytest.mark.incremental
    class Test_Scenario_2:
     def test_4(self):
       ..................
       ..................

Test case test_3() which follows after test_2() will be skipped from execution and marked as xfail as the previous test [i.e. test_2()] in the same class has failed. Selenium test automation case test_4() which is a part of class Test_Scenario_2() is also marked for incremental testing.

Here is the command for execution:

    py.test -rx --verbose --capture=no

Shown below is the execution snapshot:

![](https://cdn-images-1.medium.com/max/2668/0*Aq3VLqf1ZvHm90Ot.png)

**_Try browser [automation testing](https://www.lambdatest.com/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=webpage) on the most powerful cloud infrastructure. Leverage LambdaTest automation testing for faster, reliable and scalable experience on cloud._**

Here is the final status of the tests performed in the example:

![](https://cdn-images-1.medium.com/max/2000/0*rAxL_ass_kNfM2Oi.png)

Test_2 executes and fails whereas test_3 execution is skipped due to failure of test_2. Hence, it is marked as xfail. Though the test_4 () passes here, even if it were to fail, the test would be marked as fail (and not xfail) as there is no dependency on the test from the other class.

In case, incremental testing is removed from the above implementation, all the tests would be executed. Only test_2() will fail whereas the remaining three tests would pass as the @pytest.mark.incremental decorator is removed from the implementation.

![](https://cdn-images-1.medium.com/max/2676/0*GwWZr9KzaEW8wDgH.png)

[Pytest Tutorial: Executing Multiple Test Cases From Single File](https://www.lambdatest.com/blog/pytest-tutorial-executing-multiple-test-cases-from-single-file/?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=blog)

This certification is for professionals looking to develop advanced, hands-on expertise in Selenium automation testing with Python and take their career to the next level.

Here’s a short glimpse of the Selenium Python 101 certification from LambdaTest:

{%youtube KeSWHW4Lx1E%}

**_Experience hassle-free execution of your [Selenium Testing](https://www.lambdatest.com/selenium-automation?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=webpage) scripts by running them on the LambdaTest cloud grid. With support for over 3000 desktop and mobile environments, try LambdaTest for free and take advantage of its comprehensive testing capabilities._**

## Using maxfail Command Line Option To Stop Test Suite After N Test Failures

PyTest offers a command-line option called maxfail which is used to stop test suite after n test failures. For example, to stop the test suite after n test failures, where the value of n is 1. Once a failure is encountered, all the subsequent tests are skipped.

Though it is not a good practice to develop inter-dependent Selenium test automation cases, in some scenarios, you may have to develop such tests. The maxfail option comes handy in such cases as unnecessary execution of dependent tests can be avoided. Shown below is the syntax of maxfail:

    py.test --maxfail=<num>

## How To Skip Test In Python With pytest Using maxfail?

To demonstrate the usage of maxfail, we take the automated browser testing example which was shown in the section [How To Skip Test In Python With pytest Using @pytest.mark.incremental](https://www.lambdatest.com/blog/how-to-stop-test-suite-after-n-test-failures-in-pytest/#SkipInPython?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=blog). The implementation specific to skip the test in Python with pytest will be removed, remaining code remains the same.

**conftest.py**

    import pytest
    from selenium import webdriver
    import urllib3
    import warnings
     
    def pytest_runtest_setup(item):
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" % previousfailed.name)
            
    @pytest.fixture(scope="class")
    def driver_init(request):
        web_driver = webdriver.Chrome()
        request.cls.driver = web_driver
        yield
        web_driver.close()
        # web_driver.quit()

**test_step.py**

    import pytest
    from selenium import webdriver
    import urllib3
    import warnings
     
    def pytest_runtest_setup(item):
        previousfailed = getattr(item.parent, "_previousfailed", None)
        if previousfailed is not None:
            pytest.xfail("previous test failed (%s)" % previousfailed.name)
            
    @pytest.fixture(scope="class")
    def driver_init(request):
        web_driver = webdriver.Chrome()
        request.cls.driver = web_driver
        yield
        web_driver.close()
        # web_driver.quit()

The example shown below is executed where we stop the test suite after n test failures and the maximum number of failures set to 2. Here is the execution command:

    py.test -rx --verbose --capture=no --maxfail=2

As incremental testing is not enabled, all the four Selenium test automation cases will be executed and the execution stops once the number of failures becomes 2. Test case test_2() fails as the window title does not match the expected one, rest all the three test cases are executed. Out of 4 test cases, 1 test case fails and 3 test cases pass. Here is the execution snapshot:

![](https://cdn-images-1.medium.com/max/2576/0*bickEVZs2IpVOZ1w.png)

## All In All

In this Python tutorial, we looked at how to stop test suite after n test failures. Skip tests in Python with pytest can be followed in Selenium test automation scenarios where tests are inter-dependent and execution of dependent tests needs to be skipped in case the primary test case fails.

The @pytest.mark.incremental decorator is used to skip tests in Python with PyTest. The other mechanism to stop test suite after n test failures by using maxfail command-line option.

This brings an end to this [Python tutorial](https://www.lambdatest.com/blog/selenium-python-pytest-testing-tutorial/?utm_source=devto&utm_medium=organic&utm_campaign=feb27_bh&utm_term=bh&utm_content=blog). I hope this article was informative, and you now know skip tests in python or how to stop test suite after n test failures. In case you still have any doubts, do reach out to us in the comment section below, and we’d be happy to help you with your problem. **Happy Testing!!!**?