pytest-history 0.3.0
Pytest plugin to keep a history of your pytest runs
License: MIT
Requires Python: >=3.8,<4.0
Keywords: pytest,sqlite,sql,history

<h1 align="center">pytest-history</h1>
<p align="center">
Pytest plugin to keep history of your pytest runs
</p>

<p align="center">

<a href="https://github.com/Nicoretti/one-piece/python/pytest-history">
    <img src="https://img.shields.io/github/checks-status/nicoretti/crc/master" alt="Checks Master">
</a>
<a href="https://opensource.org/licenses/MIT">
    <img src="https://img.shields.io/pypi/l/pytest-history" alt="License">
</a>
<a href="https://pypi.org/project/pytest-history/">
    <img src="https://img.shields.io/pypi/pyversions/pytest-history" alt="Supported Python Versions">
</a>
<a href="https://pypi.org/project/pytest-history/">
    <img src="https://img.shields.io/pypi/v/pytest-history" alt="PyPi Package">
</a>
</p>

## Overview

`pytest-history` enables the tracking of test statuses and other metadata across multiple test runs, providing
additional insights into test behavior.

Initially, this plugin was developed specifically to identify potentially flaky tests (approximately 200) within a test
suite containing over 1000 tests, where various tests exhibited inconsistent behavior by failing on alternate runs.

## Purpose

- **Tracking Test History:** Capturing and storing historical test results, encompassing pass, fail, and other pertinent
  metadata.
- **Identifying Flaky Tests:** Enabling the identification of flaky tests by scrutinizing historical data, detecting
  irregularities or recurring patterns in test outcomes.
- **Facilitating Debugging:** Offering developers and testers insights into test stability, thereby assisting in
  debugging efforts and enhancing overall test reliability.

## Usage

1. Install the plugin using `pip install pytest-history`.
2. Utilize the historical data stored in the `.test-results.db` SQLite database using either the `pytest-history` CLI or an SQLite client.

## Configuration Options
### 1. INI File Setting
Users can configure the database file location by adding the following setting in their INI file:

```ini
# pytest.ini
[pytest]
pytest-history = /path/to/history.db
```

```toml
# pyproject.toml
[tool.pytest.ini_options]
pytest-history = "history.db"
```


### 2. Environment Variable
Another option to configure the database file location is through an environment variable:

Setting the environment variable in a Unix-based system:
```bash
export PYTEST_HISTORY_DB=/path/to/history.db
```

Setting the environment variable in a Windows system:
```cmd
set PYTEST_HISTORY_DB=C:\path\to\history.db
```

### 3. Command-line Parameter
Users can also specify the database file location via a command-line parameter:

```shell
pytest --history-db /path/to/history.db tests/
```


The database file path specified should be writable by the user running pytest commands.
When multiple configuration methods are used simultaneously, the command-line parameter takes precedence over the environment variable, and both take precedence over the INI file setting.


## CLI
The `pytest-history` command utilizes a SQLite database (default: `.test-results.db`) to provide analysis and information about past test runs and their results. It offers two main subcommands: `list` and `flakes`.

### Subcommands

#### `list`
- `pytest-history list` offers insights into historical data and provides two additional subcommands: `results` and `runs`.
  
##### subcommand `results`
- `pytest-history list results <id>`: Lists historic test results for a specific test run identified by its ID.
  
##### subcommand  `runs`
- `pytest-history list runs`: Lists historic test runs.

#### `flakes`
- `pytest-history flakes`: Lists all flaky tests identified in the test suite.

### Usage

```bash
pytest-history [options] <subcommand> [suboptions]

options
    -h, --help: Show the help message and exit.
    --db DB: Specify the database to be used for analyzing data (default: .test-results.db).
```

### Examples

List historic test runs
```bash
pytest-history list runs
```

List historic test results for a specific run
```bash
pytest-history list results <id>
```

List all flaky tests
```bash
pytest-history flakes
```

## Example Queries

Example: To find flaky tests between two distinct test runs, execute the following SQL query:

```sql
SELECT t1.testcase, t1.test_run, t2.test_run, t1.outcome, t2.outcome
FROM "test.results" t1
         JOIN "test.results" t2 on t1.testcase = t2.testcase AND (t1.test_run = 1 AND t2.test_run = 2)
WHERE (t1.outcome = 'passed' AND t2.outcome = 'failed')
   OR (t1.outcome = 'failed' AND t2.outcome = 'passed')
GROUP BY t1.testcase
ORDER BY t1.testcase;
```