Chapter 4: A Quick tour of PyTest
Tags: python, pytest, testing, unittesting
Published: 2022-10-21 12:14:53

- This chapter is part of the series: {% embed https://dev.to/zim95/understanding-the-python-tox-ecosystem-3noi %}
- In this section, we will be going through the features of pytest.
- Table of Contents:
    - [4.1 - Test Discovery](#part-4_1)
    - [4.2 - Parameterising our Tests with `@pytest.mark.parameterize`](#part-4_2)
    - [4.3 - Skipping Tests](#part-4_3)
    - [4.4 - Handling failing test cases](#part-4_4)
    - [4.5 - Handling tests that raise errors](#part-4_5)
    - [4.6 - `fixtures` in `pytest`](#part-4_6)
      - [4.6.1 - Understanding `fixture` scopes](#part-4_6_1)
    - [4.7 - `monkeypatching` (The correct way to mock code)](#part-4_7)
      - [4.7.1 - Understanding the need to mock](#part-4_7_1)
      - [4.7.2 - Monkeypatch syntax overview](#part-4_7_2)
    - [4.8 - Understanding Test Coverage](#part-4_8)

### 4.1 Test Discover <a name="part-4_1"></a>

- Test discovery is the process of automatically discovering tests within the specified locations.
- There are a lot more use cases of test discovery than whats mentioned in this article. We can specify the configurations on a much more granular level.
- However, we are going to look at the default discovery mechanism that we currently have in `pytest`.
- The default implementation is as follows:
    - It looks at the tests in our `tests/` directory. We specified this directory in the `tool.pytest.ini` option in `pyproject.toml` file.
    - It then looks for any module (filename in this case) that starts with `test_`.
    - Within each module it looks for functions with `test_`, it assumes that these are the tests and it runs them.
- To know more about `pytest` test discovery, we can read the documentation.
    - Here is a small article: [https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery](https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery)
    - Here is an article on how to change the standard test discovery: [https://docs.pytest.org/en/7.1.x/example/pythoncollection.html](https://docs.pytest.org/en/7.1.x/example/pythoncollection.html)

### 4.2 Parameterising our Tests <a name="part-4_2"></a>
- For now, our code has a single test function inside `test_app.py`.
    
    ```python
    import pytest
    from app import do_something_with_somethingelse, SomethingElse
    
    def test_something() -> None:
    	somethingelse1: SomethingElse = SomethingElse(1)
    	somethingelse2: SomethingElse = SomethingElse(2)
    	somethingelse3: SomethingElse = SomethingElse(3)
    	assert do_something_with_somethingelse(somethingelse1) == 'some'
    	assert do_something_with_somethingelse(somethingelse2) == 'thing'
    	assert do_something_with_somethingelse(somethingelse3) == ''
    ```
    
- This test does not indicate the behaviour that we are testing. But later on, we will have multiple tests based on the different tests.
- Not only that, notice that in the tests, if one `assert` statement fails. Then the rest of the `assert` statements are never executed. The test is considered to have failed.
- In order to prevent this from happening, `pytest` allows `parameterization` of `tests`.
- It has a `@pytest.mark.parameterize` decorator using which we can plug in different values for the same use case (same test function).
- This allows our tests to run multiple `assert` statements. If one fails, the others will still run.
- Using this our test could look something like:
    
    ```python
    import pytest
    from app import do_something_with_somethingelse, SomethingElse
    
    @pytest.mark.parameterize("dummy_somethingelse_value, dummy_expected_result", [
    	(1, 'some'),
    	(2, 'thing'),
    	(3, '')
    ])
    def test_something(dummy_somethingelse_value, dummy_expected_result) -> None:
    	assert do_something_with_somethingelse(SomethingElse(dummy_expected_result)) == \
    		dummy_expected_result
    ```
    
- The syntax for `parameterize` is:
    
    ```markdown
    @pytest.mark.parameterize('label1, label2, ... and so on', [
    	(label1value, label2value, ... and so on),
    	(label1value, label2value, ... and so on),
    	...
    	and so on
    ])
    def test_dummy(label1, label2, .... and so on) -> None:
    	pass
    ```
    
- So the main idea behind `mark.parameterize` is that if one of the cases in `parameterize` list fails, it still runs the rest of them.
- Apart from that, it makes our tests look a lot cleaner.

### 4.3 Skipping Tests <a name="part-4_3"></a>

- We can skip tests in `pytest` by using the `@pytest.mark.skip(reason='Any message')` decorator.
    
    ```python
    @pytest.mark.skip(reason='Feature not implemented yet')
    def test_skip_sometest() -> None:
    	assert some_uninmplemted_feature is not None
    ```
    
- When we do `TDD (Test Driven Development)`, we write tests before we implement the functionality.
- We usually skip such tests.
- There is also another `skipif` variation of skip which lets us skip based on conditions. For example, if the operating system is Windows, or the version of Python is not supported, etc.

### 4.4 Handling Failing Tests <a name="part-4_4"></a>

- We might have tests that are bound to fail. These tests can be used as:
    - Tests to assure that things that are supposed to fail (like some virtual scenario), do fail.
    - Documentation to better understand the system. To show that certain settings will fail.
- These tests when executed will show up as failure and fail the entire build.
- We can avoid that by deliberately marking them as tests that are supposed to fail.
- We can achieve this with the `pytest.mark.xfail` decorator.
    
    ```python
    @pytest.mark.xfail
    def test_zero_division() -> None:
    	assert 1 / 0 == 1
    ```
    
- However, we should not mark those tests that raise errors as failing tests. This brings us to the next topic.

### 4.5 Handling Tests that raise Exceptions <a name="part-4_5"></a>

- We might want to make sure certain scenarios raise a certain exceptions.
- This can be handled by using the context manager called `pytest.raises(Exception)`.
    
    ```python
    def zero_division() -> float:
    	return 1/0
    
    def test_zero_division() -> None:
    	with pytest.raises(ZeroDivisionError):
    		zero_division()
    ```
    
- As we can see, there is an exception expected when we run `zero_division`. We can handle it with `pytest.raises`.
- This test will now fail if `ZeroDivisionError` is not raised.

### 4.6 `fixtures` in `pytest` <a name="part-4_6"></a>

- Fixtures are used when multiple tests require a certain amount of setup that they all share in common.
- For example, tests that require a database connection.
    - In such cases, it is better to create the connection and share it with tests.
    - Rather than creating a connection for every test.
    - So we can create `Fixtures` that get passed to our tests.
- In `unittest` this is done by the `setUp` method. It creates objects that can be shared throughout the test methods.
- By convention if we create a file called `conftest.py`, and create a fixture in that file, it will be available for all of our tests based on the`scope` of the fixture.
- So we create `conftest.py` in our `tests/` directory alongside `test_app.py` and create fixtures there as follows:
    
    ```python
    import pytest
    
    @pytest.fixture(scope="session")
    def db_con():
    	db_url = "driver+dialect://username:password@host:port/database"
    	db = DatabaseLibrary()
    	with db.connection(db_url) as conn:
    		yield conn
    ```
- Notice we have scopes in `pytest`. We will look at what scopes are.
- Fixture scopes basically define how many times a fixture will run based on the area it covers.

### 4.6.1 Understanding `fixture` scopes <a name="part-4_6_1"></a>
- To learn more about `fixture` `scope`, we can go: [https://betterprogramming.pub/understand-5-scopes-of-pytest-fixtures-1b607b5c19ed](https://betterprogramming.pub/understand-5-scopes-of-pytest-fixtures-1b607b5c19ed).
- Scope basically has to do with the scope of the fixture as in, upto where the fixture is shared. There are five types of scopes: `function`, `class`, `module`, `package`, `session`.
- `function` scope:
    - This is the default scope without explicitly adding `scope='function'`.
    - The fixture will be executed per test function.
    - This is very heavy task.
    - This fixture is suitable for single use functions.
        ```python
        import pytest
        
        @pytest.fixture()
        def only_used_once():
            with open("app.json") as f:
                config = json.load(f)
            return config
        ```
                
    - This fixture is suitable when it handles very lightweight operations such as returning a constant or a different value every time.
        ```python
        import pytest
        from datetime import datetime
        
        @pytest.fixture()
        def light_operation():
            return "I'm a constant"
        
        @pytest.fixture()
        def need_different_value_each_time():
            return datetime.now()
        ```
                
- `class` scope:
    - This scope runs the fixture per test class.
    - If we have couple of test functions that do similar things, we can put them in the same class.
    - To know more about grouping tests in classes.
        - Conventions for Python Test Discovery: [https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery](https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery)
        - Group multiple tests in a class: [https://docs.pytest.org/en/stable/getting-started.html#group-multiple-tests-in-a-class](https://docs.pytest.org/en/stable/getting-started.html#group-multiple-tests-in-a-class)
        - The parent StackOverFlow thread: [https://stackoverflow.com/questions/20277058/py-test-does-not-find-tests-under-a-class#:~:text=test does not find tests under a class,-Ask Question](https://stackoverflow.com/questions/20277058/py-test-does-not-find-tests-under-a-class#:~:text=test%20does%20not%20find%20tests%20under%20a%20class,-Ask%20Question)
        - How to test `unittest.TestCase` based classes: [https://docs.pytest.org/en/7.1.x/how-to/unittest.html](https://docs.pytest.org/en/7.1.x/how-to/unittest.html)
        - Changing Python standard test discovery: [https://docs.pytest.org/en/7.1.x/example/pythoncollection.html](https://docs.pytest.org/en/7.1.x/example/pythoncollection.html)
    - Now that we know how to group tests in classes and test discovery for classes, we will see how we can use fixtures on these test classes with `scope="class"`.
    - We can use `@pytest.mark.usefixtures("fixturename")` decorator with the test class.
        ```python
        @pytest.fixture(scope="class")
        def dummy_data(request):
            request.cls.num1 = 10
            request.cls.num2 = 20
            logging.info("Execute fixture")
        
        @pytest.mark.usefixtures("dummy_data")
        class TestCalculatorClass:
            def test_distance(self):
                logging.info("Test distance function")
                assert distance(self.num1, self.num2) == 10
        
            def test_sum_of_square(self):
                logging.info("Test sum of square function")
                assert sum_of_square(self.num1, self.num2) == 500
        ```
    - The fixture will be executed once per test class before the test methods.
- There is a special usage of `yield` statement in `pytest` that allows us to run the fixture after all the test functions.
    - The code before `yield` acts as `setup code`.
    - The code after `yeild` acts as `teardown code`.
    - For example, testing a database.
        ```python
        @pytest.fixture(scope="class")
        def prepare_db(request):
            # pseudo code
            connection = db.create_connection()
            request.cls.connection = connection
            yield
            connection = db.close()
        
        @pytest.mark.usefixtures("prepare_db")
        class TestDBClass:
            def test_query1(self):
                assert self.connection.execute("..") == "..."
        
            def test_query2(self):
                assert self.connection.execute("..") == "..."
        ```
    - We can `yield` values to any test that wants it and the remaining code will act as the `teardown code`.
- `module and package` scopes:
    - The `scope='module'` runs the fixture per module (per file).
        - A module may contain multiple functions as well as classes.
        - No matter how many tests are in the module, the fixture is run only once.
    - The `scope='package'` runs the fixture per package (directory).
        - A package contains one or more modules.
        - No matter how many modules there are, the fixture is run only once.
    - An example would be:
        
        ```python
        @pytest.fixture(scope="module")
        def read_config():
            with open("app.json") as f:
                config = json.load(f)
                logging.info("Read config")
            return config
        ```
        
    - We might want to read config only once per module or only once throughout the entire package.
- `session` scope:
    - Every time we run `pytest`, it is considered to be one session.
    - The `scope='session'` makes sure the fixture only executes per session.
    - Session scopes are designed for expensive operations like truncating a table or loading a test set to a database.
- We can read more about fixture topics such as `autouse` and execution order of fixtures. We will look at them when required.

### 4.7 Monkey Patching in PyTest <a name="part-4_7"></a>
- `Monkey patching` is the process of changing the declaration of code during runtime.
- This is `mocking` done right.
- To understand `mocking` and `test-driven development` in `Python`, you can check out my tutorial series on `TDD in Python`: [https://www.youtube.com/watch?v=NAjCDS-qxrQ&list=PLVWk3kKbCAAuPPr_eh4KNKdL2hDSNFH3H](https://www.youtube.com/watch?v=NAjCDS-qxrQ&list=PLVWk3kKbCAAuPPr_eh4KNKdL2hDSNFH3H).
- `TLDW(Too long didn’t watch?)`: Mocking lets us replace methods with dummy objects during runtime.

### 4.7.1 Understanding the need to mock <a name="part-4_7_1"></a>
- Why would we want to mock anything?
- To understand this you need to understand the principals of unit testing:
    - Unit testing are the fastest tests. Why?
    - It is testing the functionality of a single unit, without any external dependencies.
    - Say you test a function which calls the database.
    - You would only want to test the flow of such a function without making the call to the actual database.
    - How do you do that? You basically, `mock` the entire database connection object and replace it with a dummy object.
    - You can set whatever values you want that dummy object to return, create whatever dummy functions are required along with their return types. You can mimic the entire database connection’s set of methods if you want.
    - What this allows us to do is set dummy values for all the external calls that are within our function and set default results for them.
    - With this we can setup a fake scenario without actually making external calls.
    - This is why unit tests are fast.
    - Integration testing is on the other hand is testing of the actual values in which case, we would test the actual results from the external calls and evaluate them. Thats why Integration tests are slower than unit tests.
- This is one of the reasons we use mock.
- `Monkey patching` gives us a better interface than barebones `mock` library.

### 4.7.2 Monkeypatch syntax overview <a name="part-4_7_2"></a>
- `Fixtures` can also depend on other `fixtures`. `monkeypatching` is such a fixture which is available to all functions.
- Let’s create a `fixture`:
    
    ```python
    import pytest
    
    @pytest.fixture(scope="function")
    def capture_stdout(monkeypatch)
    	std_output: dict = {'output': '', 'write_count': 0}
    	def fake_writer(s) -> dict:
    		std_output['output'] += s
    		std_output['write_count'] += 1
            return std_output
    	
    	monkeypatch.setattr(sys.stdout, 'write', fake_writer)
    	return std_output
    
    def test_print(capture_stdout):
    	print("Hello")
    	assert capture_stdout['output'] == "Hello"
    ```
- What we did here is replace `sys.stdout.write` with `fake_writer` which returns the captured `dictionary`.
- Every time a test function uses `capture_stdout`  fixture, the environment within the test changes where `sys.stdout.write` is replaced with `fake_writer`.
- Whenever any function calls `sys.stdout` within the function scope will be making a call to `fake_writer` instead. In our example, we call `print` will calls to `sys.stdout`.
- The return value is captured in the `capture_stdout` variable and can be used to `assert` results.
- This is how we `mock` things.
- But why use `monkeypatching` instead of just mocks?
    - At the end of the test, `monkeypatching` makes sure that everything is undone and `sys.out.writer` goes back to its original value.
    - In `unittest` we use `mock.patch` as `decorators` and `context managers`. This also  makes sure that everything is undone at the end of the test.

### 4.8 UNDERSTANDING TEST COVERAGE <a name="part-4_8"></a>

- We have put `--cov` options in `pyproject.toml`.
- This shows us test coverage in percentages.
- Here `100%` test coverage means all our tests combined and touched every single line of executable code in the `src`.
- Any percentage lesser means that some executable code was not tested by our tests.
- However, having `100%` test coverage does denote that there are no bugs. It all depends. However, it’s good to have `100%` code coverage.