regression-testing-utils 0.1.2
Collection of Python tools to support execution of regression testing.
Requires Python: >=3.10
Keywords: regression_testing_utils

========================
Regression Testing Utils
========================

Collection of Python tools to support execution of regression testing.


Exported Console Scripts
------------------------

The following exported console scripts are available:

- regression-testing-runner


Usage
-----

Install the package:

.. code-block:: shell

    pip install regression-testing-utils


Prepare your regression configuration file.

Contents of example.yaml:

.. code-block:: yaml

    ---
    # Acceptable values for file_format are csv or tsv.
    # Meaning, this software can only compare files
    # where both files are tab-delimited (tsv) or both
    # files are comma-separated (csv).
    file_format: tsv

    # The line number that the column headers occur on.
    # Column headers are necessary for reporting which
    # which columns the differences were detected in.
    header_line_number: 1

    # The line number that the records start on.
    # Beginning on this line, records will be compared
    # between the two files.
    records_start_line_number: 2

    # The maximum number of columns.  If specified, this
    # will inform the software up to which line should
    # be consider. Otherwise, all lines beginning at-
    # and including records_start_line_number (above).
    max_column_count:

    # If the following is True, then the evaluator software will ignore
    # the columns listed in the "ignore_columns_str".
    ignore_columns: false

    # The following will list the columns to be ignored by the evaluator
    # software.  Note: if the evaluator software should ignore columns
    # specified in this list, then the "ignore_columns" boolean above
    # should be set to true.
    ignore_columns_str: null

    workflow:
      # If running in test mode, the actual commands will not be executed.
      test_mode: false
      stages:
        # Each workflow stage is optional.
        # The stage_name and commands are all required.
        # The desc (description) is optional.
        - stage_name: evaluation
          desc: "Evaluate the results of the software being tested/evaluated"
          commands:
            - evaluator:
              # Set-up for testing:
              # cp regression-testing-utils/examples/example_actual.tsv /tmp/demo-regression-testing-utils/actual.tsv
              # cp regression-testing-utils/examples/example_expected.tsv /tmp/demo-regression-testing-utils/expected.tsv
              args:
                # file1 is required.  It must indicate the first file that will be compared against file2.
                file1: /tmp/demo-regression-testing-utils/actual.tsv
                # file2 is required.  It must indicate the first file that will be compared against file1.
                file2: /tmp/demo-regression-testing-utils/expected.tsv
                # file_format is optional.  It indicates what the file formats are.
                file_format: tsv
                # outfile is required.  It indicates where the comparison report will be written to.
                outfile: /tmp/demo-regression-testing-utils/comparison_report.txt
    # If the following is set to true,
    # the software will use the Rich Table
    # API for displaying which stage is
    # being executed.
    # If set to false, will use a simple
    # banner to display the same information.
    use_rich_table: true

Contents of actual.tsv

.. code-block:: text

    cat -n /tmp/demo-regression-testing-utils/actual.tsv             
    1  #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
    2  1       12345   rs567   A       G       50      PASS    DP=30;AF=0.2;AN=1000;CSQ=missense_variant|HIGH|GeneA|ENSG00000112345|transcriptA|ENST00000234567|protein_coding|1/10|c.123C>T|p.Arg41Trp|123/1000|ensembl
    3  2       56789   rs555   T       C       44      PASS    DP=25;AF=0.1;AN=1200;CSQ=synonymous_variant|MEDIUM|GeneB|ENSG00000123456|transcriptB|ENST00000345678|protein_coding|5/20|c.567A>G|p.Ala189Ala|567/1200|ensembl
    4  3       98765   rs123   G       T       61      PASS    DP=40;AF=0.3;AN=800;CSQ=splice_acceptor_variant|HIGH|GeneC|ENSG00000134567|transcriptC|ENST00000456789|protein_coding|2/15|c.987+1G>T|p.?|987/800|ensembl
    5  1       34567   rs456   C       A       55      FAIL    DP=35;AF=0.15;AN=900;CSQ=frameshift_variant|HIGH|GeneX|ENSG00000145678|transcriptX|ENST00000567890|protein_coding|8/25|c.345_346insT|p.Leu116Phefs*12|345/900|ensembl

Contents of expected.tsv

.. code-block:: text

    cat -n /tmp/demo-regression-testing-utils/expected.tsv 
    1  #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
    2  1       12345   rs567   A       G       50      PASS    DP=30;AF=0.2;AN=1000;CSQ=missense_variant|HIGH|GeneA|ENSG00000112345|transcriptA|ENST00000234567|protein_coding|1/10|c.123C>T|p.Arg41Trp|123/1000|ensembl
    3  2       56789   rs890   T       C       44      PASS    DP=25;AF=0.1;AN=1200;CSQ=synonymous_variant|MEDIUM|GeneB|ENSG00000123456|transcriptB|ENST00000345678|protein_coding|5/20|c.567A>G|p.Ala189Ala|567/1200|ensembl
    4  3       98765   rs123   G       T       60      PASS    DP=40;AF=0.3;AN=800;CSQ=splice_acceptor_variant|HIGH|GeneC|ENSG00000134567|transcriptC|ENST00000456789|protein_coding|2/15|c.987+1G>T|p.?|987/800|ensembl
    5  1       34567   rs456   C       A       55      PASS    DP=35;AF=0.15;AN=900;CSQ=frameshift_variant|HIGH|GeneX|ENSG00000145678|transcriptX|ENST00000567890|protein_coding|8/25|c.345_346insT|p.Leu116Phefs*12|345/900|ensembl

Execution of exported console script:

.. code-block:: shell

    regression-testing-runner --config_file /tmp/demo-regression-testing-utils/simple.yaml --outdir /tmp/demo-regression-testing-utils
    --logfile was not specified and therefore was set to '/tmp/demo-regression-testing-utils/runner.log'
    Setting file format to 'tsv'
    3 differences found

Contents of comparison report:

.. code-block:: text

    cat -n /tmp/demo-regression-testing-utils/comparison_report.txt   
     1  ## method-created: /tmp/regression-testing-utils/venv/lib/python3.10/site-packages/regression_testing_utils/evaluator.py
     2  ## date-created: 2025-02-15-121022
     3  ## created-by: sundaram
     4  ## file 1: /tmp/demo-regression-testing-utils/actual.tsv
     5  ## file 2: /tmp/demo-regression-testing-utils/expected.tsv
     6  ## file-format: tsv
     7  ## logfile: /tmp/demo-regression-testing-utils/runner.log
     8  ## Number of differences: 3
     9  Line #  Column Name     Column #        Column Letter   Value in File 1 Value in File 2
    10  1       rs567   3       C       rs555   rs890
    11  2       50      6       F       61      60
    12  3       PASS    7       G       FAIL    PASS


=======
History
=======

0.1.0 (2024-02-24)
------------------

* First release on PyPI.