Using Control and Test Groups in Tests
Tags: testing, test, beginners, programming
Published: 2023-10-22 19:13:16

Computer Science degrees at any university will focus a lot on mathematically proving things; after all, it is a science. But then, in the real world, most jobs don't require you to come up with new complex algorithms and mathematically prove them. Instead, most of the time, you only need to write tests to prove that your code works. The problem is that sometimes, these tests are not rigorous enough.

## Well designed experiments
Consider a study where you need to check if a pill cures an illness. As a simple experiment, you could use two groups of patients; some receive a placebo (control group), and the others receive the real pill (test group). They don't know which pill they took, and the only difference between the two groups is the pill. If the pill works, there would be a considerable difference in the number of patients recovered in the test group vs the control group. However, if the pill doesn't work, then the outcome in both groups will be pretty much the same, or people in the test group could be even sickier 🫣. We need a control group because, otherwise, how would you know that the people who got the pill recovered because of it and nothing else?

## Writing tests as experiments
Consider, for example, a Website where there is an admin url that only users with admin permissions can access, and anyone else would see a 404.

You could quickly create a test like this:
```python
def test_admin_access():
    regular_user = make_user()
    client = HTTPClient()
    response = client.get("http://localhost:1234/admin")
    assert response.status_code == 404
```

As per the pills and patient example, how do you known the 404 code is not due to something else? In fact, the user is not logged in, which could be causing a 404. And what if the URL is wrong? Wouldn't that also cause a 404? So, this test can pass for the wrong reasons, and the admin endpoint could inadvertently allow access to regular users.

Well, let's change it a little bit:

```python
def test_regular_user_cannot_access():
    regular_user = make_user()

    client = HTTPClient()
    client.login(regular_user)
    
    response = client.get("http://localhost:1234/admn")
    assert response.status_code == 404

def test_admin_user_can_access():
    admin_user = make_user(admin=True)

    client = HTTPClient()
    client.login(regular_user)
    
    response = client.get("http://localhost:1234/admin")
    assert response.status_code == 200
```

**WRONG!!!**

While it looks like we added a control group at first sight, we are not performing the same experiment in both. Note the typo in the URL for the first test; the request will return 404 because the URL is wrong and the test will "pass".

Fixing the typo is not a real fix. The right way to fix this is to separate the experiment into its own function and call it for each user type.

```python
def get_admin_page(user):  # experiment
    client = HTTPClient()
    if user is not None:
        client.login(user)
    return client.get("http://localhost:1234/admin")

def test_admin_access():
    regular_user = make_user()
    admin_user = make_user(admin=True)
    
    # Test
    no_user_response = get_admin_page(None)
    regular_user_response = get_admin_page(regular_user)
    # Control
    admin_user_response = get_admin_page(admin_user)
    
    # Test
    assert no_user_response.status_code == 404
    assert regular_user_response.status_code == 404
    # Control
    assert admin_user_response.status_code == 200
```

Bonus: we are also testing that if the user hasn't logged in, it can reuse the same control group.

While I have added them under the same test function, it doesn't matter if you break it into multiple tests as long as they all use `get_admin_page`.

We were able to test it as if it was an experiment because the users are objects, where we change an independent variable, i.e. `is_admin` or `is_logged_in`, and we check that the experiment (`get_admin_page`) returns the expected result for each.

## When it doesn't apply
There are some cases where you need to resort to other methods. Consider the following function:

```python
def get_percentage(portion, total):
    portion/total * 100
```

You could try and come up with a control and test group, but under which criteria are you separating the groups? Both portion and total are independent variables, but we are not performing an experiment over an object. A different way could be proving it mathematically, though it is a simple formula. So, perhaps the best way to ensure that you didn't type `101` instead of `100` is to write a simple test that checks that the result is as expected for certain inputs.