pytest-github 0.3.1
Plugin for py.test that associates tests with github issues using a marker.
License: MIT
Keywords: py.test pytest testing github issues

pytest-github
=============

`Build Status <https://travis-ci.org/jlaska/pytest-github>`__ `Coverage
Status <https://coveralls.io/r/jlaska/pytest-github>`__ `Requirements
Status <https://requires.io/github/jlaska/pytest-github/requirements/?branch=master>`__
`Version <https://pypi.python.org/pypi/pytest-github/>`__
`License <https://pypi.python.org/pypi/pytest-github/>`__ `Supported
Python Versions <https://pypi.python.org/pypi/pytest-github/>`__

Plugin for py.test that integrates with github using markers.
Integration allows tests to xfail (or skip) based on the status of
linked github issues.

Installation
------------

Install the plugin using ``pip``

.. code:: bash

   pip install pytest-github

Usage
-----

1. Once installed, the following ``py.test`` command-line parameters are
   available.

.. code:: bash

   py.test \
       [--github-cfg=GITHUB_CFG] \
       [--github-username=GITHUB_USERNAME] \
       [--github-token=GITHUB_TOKEN] \
       [--github-completed=GITHUB_COMPLETED] \
       [--github-summary]

2. Next, create a configure file called ``github.yml`` that contains
   your GitHub username and `personal api
   token <https://github.com/blog/1509-personal-api-tokens>`__. A sample
   file is included below.

.. code:: yaml

   github:
       username: j.doe
       token: XXXXXXXXXXXXX

Marker
~~~~~~

The following ``py.test`` marker is available:

.. code:: python

   @pytest.mark.github(*args): GitHub issue integration

The marker can be used to influence the outcome of tests. See the
examples below for guidance.

Example: xfail
~~~~~~~~~~~~~~

Often, when a test fails, one might file a GitHub issue to track the
resolution of the problem. Alternatively, you could use the built-in
``xfail`` marker. This is where ``pytest-github`` can be of use. To
avoid having to review known failures with each test run, and to avoid
always using ``xfail``, consider the ``github`` marker to dynamically
influence the test outcome based on the state of the GitHub issue.

The following example demonstrates using the ``github`` marker to
influence the outcome of a known failing test.

.. code:: python

   @pytest.mark.github('https://github.com/some/open/issues/1')
   def test_will_xfail():
       assert False

Running this test with ``py.test`` will produce the following output:

.. code:: bash

   test.py::test_will_xfail xfail

Example: Anticipating specific exceptions with the ‘raises’ keyword
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To avoid masking additional failures that might be uncovered by a test
while a github issue is being resolved, you can restrict expected
failures to specific exceptions using the ``raises`` keyword argument:

.. code:: python

   @pytest.mark.github('https://github.com/some/open/issues/1', raises=ZeroDivisionError)
   def test_will_xfail():
       foo = 1/0


   @pytest.mark.github('https://github.com/some/open/issues/1', raises=ValueError)
   def test_will_fail():
       # This test has been marked with an open issue but it will still fail
       # because the exception raised is different from the one indicated by
       # the 'raises' keyword.
       foo = 1/0

Running this test with ``py.test`` will produce the following output:

.. code:: bash

   collected 2 items
   collected 1 github issues

   test.py::test_will_xfail xfail
   test.py::test_will_fail FAILED

Example: XPASS
~~~~~~~~~~~~~~

The following example demonstrates a test that succeeds, despite being
associated with an *open* GitHub issue.

.. code:: python

   @pytest.mark.github('https://github.com/some/open/issues/1')
   def test_will_xpass():
       assert True

In this example, the ``XPASS`` outcome (a.k.a. unexpected pass) is used.

::

   test.py::test_will_xpass XPASS

Example: PASSED
~~~~~~~~~~~~~~~

The following example demonstrates a test that succeeds, while it is
associated with a *closed* GitHub issue. When a test associated with a
GitHub

.. code:: python

   @pytest.mark.github('https://github.com/some/closed/issues/2')
   def test_will_pass():
       assert True

In this example, the ``PASSED`` outcome is used.

::

   test.py::test_will_pass PASSED

Example: FAILED
~~~~~~~~~~~~~~~

The following example demonstrates a test that fails, while it is
associated with a *closed* GitHub issue.

.. code:: python

   @pytest.mark.github('https://github.com/some/closed/issues/2')
   def test_will_fail():
       assert False

In this example, the ``FAILED`` outcome is used.

::

   test.py::test_will_fail FAILED

Example: SKIPPED
~~~~~~~~~~~~~~~~

The following example demonstrates a test that fails, while it is
associated with an *open* GitHub issue.

.. code:: python

   @pytest.mark.github('https://github.com/some/open/issues/1', skip=True)
   def test_will_skip():
       assert False

In this example, the ``SKIPPED`` outcome is used.

::

   test.py::test_will_skip SKIPPED

In this example, the ``SKIPPED`` outcome is used.

::

   test.py::test_will_skip SKIPPED

Example: IDS
~~~~~~~~~~~~

The following example demonstrates a parametrize test that uses the
``github`` marker to influence the outcome of a subset of the known
failing test.

.. code:: python

   @pytest.mark.github('https://github.com/some/open/issues/1', ids=['even2', 'even4'])
   @pytest.mark.parametrize("count", [1, 2, 3, 4], ids=["odd1", "even2", "odd3", "even4"])
   def test_will_xfail(count):
       assert count % 2

Summary of GitHub markers and their associated tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``--github-summary`` option lists all GitHub issues referenced by a
``github`` marker. The list is divided into two sections,
``Resolved Issues`` and ``Unresolved Issues``, where an issue is
considered resolved if it has one of the ``GITHUB_COMPLETED`` labels.
Beneath each issue is a listing of all tests that reference the issue.

Sample output:

::

   Unresolved Issues
   https://github.com/repo/open/issues/1
    - test_suite.py:test_foo
   https://github.com/repo/open/issues/2
    - test_suite.py:test_bar

   Resolved Issues
   https://github.com/repo/open/issues/3
    - test_suite.py:test_baz
    - test_suite.py:test_bah