plone.testing 10.0.0a1
Testing infrastructure for Zope and Plone projects.
License: BSD
Requires Python: >=3.10
Keywords: plone zope testing

Introduction
============

.. contents:: Table of contents

``plone.testing`` provides tools for writing unit and integration tests in a Zope and Plone environment.
It is not tied to Plone, and it does not depend on Zope (although it has some optional Zope-only features).

``plone.testing`` builds on `zope.testing`_, in particular its layers concept.
This package also aims to promote some "good practice" for writing tests of various types.

.. note::

   If you are working with Plone, there is a complementary package `plone.app.testing`_, which builds on ``plone.testing`` to provide additional layers useful for testing Plone add-ons.

If you are new to automated testing and test driven development, you should spend some time learning about those concepts.
Some useful references include:

* `The Wikipedia article on unit testing <https://en.wikipedia.org/wiki/Unit_testing>`_
* `The Dive Into Python chapter on testing <https://diveintopython3.problemsolving.io/unit-testing.html>`_

Bear in mind that different Python frameworks have slightly different takes on how to approach testing.
Therefore, you may find examples that are different to those shown below.
The core concepts should be consistent, however.

Compatibility
-------------

``plone.testing`` 7.x has been tested with Python 2.7 and 3.6.
If you're using the optional Zope layers, you must use Zope version 4 or later.
Look at older ``plone.testing`` versions for supporting older Zope versions.

Definitions
-----------

In this documentation, we will use a number of testing-related terms.
The following definitions apply:

Unit test
    An automated test (i.e. one written in code) that tests a single unit (normally a function) in isolation.
    A unit test attempts to prove that the given function works as expected and gives the correct output given a particular input.
    It is common to have a number of unit tests for a single function, testing different inputs, including boundary cases and errors.
    Unit tests are typically quick to write and run.

Integration test
    An automated test that tests how a number of units interact.
    In a Zope context, this often pertains to how a particular object or view interacts with the Zope framework, the ZODB persistence engine, and so on.
    Integration tests usually require some setup and can be slower to run than unit tests.
    It is common to have fewer integration tests than unit test.

Functional test
    An automated test that tests a feature in an "end-to-end" fashion.
    In a Zope context, that normally means that it invokes an action in the same way that a user would, i.e. through a web request.
    Functional tests are normally slower to run than either unit or integration tests, and can be significantly slower to run.
    It is therefore common to have only a few functional tests for each major feature, relying on unit and integration tests for the bulk of testing.

Black box testing
    Testing which only considers the system's defined inputs and outputs.
    For example, a functional test is normally a black box test that provides inputs only through the defined interface (e.g. URLs published in a web application), and makes assertions only on end outputs (e.g. the response returned for requests to those URLs).

White box testing
    Testing which examines the internal state of a system to make assertions.
    Authors of unit and integration tests normally have significant knowledge of the implementation of the code under test, and can examine such things as data in a database or changes to the system's environment to determine if the test succeeded or failed.

Assertion
    A check that determines whether a test succeeds or fails.
    For example, if a unit test for the function ``foo()`` expects it to return the value 1, an assertion could be written to verify this fact.
    A test is said to *fail* if any of its assertions fail.
    A test always contains one or more assertions.

Test case
    A single unit, integration or functional test.
    Often shortened to just *test*.
    A test case sets up, executes and makes assertions against a single scenario that bears testing.

Test fixture
    The state used as a baseline for one or more tests.
    The test fixture is *set up* before each test is executed, and *torn down* afterwards.
    This is a pre-requisite for *test isolation* - the principle that tests should be independent of one another.

Layer
    The configuration of a test fixture shared by a number of tests.
    All test cases that belong to a particular layer will be executed together.
    The layer is *set up* once before the tests are executed, and *torn down* once after.
    Layers may depend on one another.
    Any *base layers* are set up before and torn down after a particular *child layer* is used.
    The test runner will order test execution to minimise layer setup and tear-down.

Test suite
    A collection of test cases (and layers) that are executed together.

Test runner
    The program which executes tests.
    This is responsible for calling layer and test fixture set-up and tear-down methods.
    It also reports on the test run, usually by printing output to the console.

Coverage
    To have confidence in your code, you should ensure it is adequately covered by tests.
    That is, each line of code, and each possible branching point (loops, ``if`` statements) should be executed by a test.
    This is known as *coverage*, and is normally measured as a percentage of lines of non-test code covered by tests.
    Coverage can be measured by the test runner, which keeps track of which lines of code were executed in a given test run.

Doctest
    A style of testing where tests are written as examples that could be typed into the interactive Python interpreter.
    The test runner executes each example and checks the actual output against the expected output.
    Doctests can either be placed in the docstring of a method, or in a separate file.
    The use of doctests is largely a personal preference.
    Some developers like to write documentation as doctests, which has the advantage that code samples can be automatically tested for correctness.
    You can read more about doctests on `Wikipedia <http://en.wikipedia.org/wiki/Doctest>`_.

Installation and usage
======================

To use ``plone.testing`` in your own package, you need to add it as a dependency.
Most people prefer to keep test-only dependencies separate, so that they do not need to be installed in scenarios (such as on a production server) where the tests will not be run.
This can be achieved using a ``test`` extra.

In ``setup.py``, add or modify the ``extras_require`` option, like so:::

    extras_require = {
        'test': [
                'plone.testing',
            ]
    },

You can add other test-only dependencies to that list as well, of course.

To run tests, you need a test runner.
If you are using ``zc.buildout``, you can install a test runner using the `zc.recipe.testrunner`_ recipe.
For example, you could add the following to your ``buildout.cfg``:::

    [test]
    recipe = zc.recipe.testrunner
    eggs =
        my.package [test]
    defaults = ['--auto-color', '--auto-progress']

You'll also need to add this part to the ``parts`` list, of course:::

    [buildout]
    parts =
        ...
        test

In this example, have listed a single package to test, called ``my.package``, and asked for it to be installed with the ``[test]`` extra.
This will install any regular dependencies (listed in the ``install_requires`` option in ``setup.py``), as well as those in the list associated with the ``test`` key in the ``extras_require`` option.

Note that it becomes important to properly list your dependencies here, because the test runner will only be aware of the packages explicitly listed, and their dependencies.
For example, if your package depends on Zope, you need to list ``Zope`` in the ``install_requires`` list in ``setup.py``;
ditto for ``Plone``, or indeed any other package you import from.

Once you have re-run buildout, the test runner will be installed as ``bin/test`` (the executable name is taken from the name of the buildout part).
You can execute it without arguments to run all tests of each egg listed in the ``eggs`` list::

    $ bin/test

If you have listed several eggs, and you want to run the tests for a particular one, you can do::

    $ bin/test -s my.package

If you want to run only a particular test within this package, use the ``-t`` option.
This can be passed a regular expression matching either a doctest file name or a test method name.::

    $ bin/test -s my.package -t test_spaceship

There are other command line options, which you can find by running::

    $ bin/test --help

Also note the ``defaults`` option in the buildout configuration.
This can be used to set default command line options.
Some commonly useful options are shown above.

Coverage reporting
------------------

When writing tests, it is useful to know how well your tests cover your code.
You can create coverage reports via the excellent `coverage`_ library.
In order to use it, we need to install it and a reporting script::

    [buildout]
    parts =
        ...
        test
        coverage
        report

    [coverage]
    recipe = zc.recipe.egg
    eggs = coverage
    initialization =
        include = '--source=${buildout:directory}/src'
        sys.argv = sys.argv[:] + ['run', include, 'bin/test', '--all']

    [report]
    recipe = zc.recipe.egg
    eggs = coverage
    scripts = coverage=report
    initialization =
        sys.argv = sys.argv[:] + ['html', '-i']

This will run the ``bin/test`` script with arguments like `--all` to run all layers.
You can also specify no or some other arguments.
It will place coverage reporting information in a ``.coverage`` file inside your buildout root.
Via the ``--source`` argument you specify the directories containing code you want to cover.
The coverage script would otherwise generate coverage information for all executed code, including other packages and even the standard library.

Running the ``bin/report`` script will generate a human readable HTML representation of the run in the `htmlcov` directory.
Open the contained `index.html` in a browser to see the result.

If you want to generate an XML representation suitable for the `Cobertura`_ plugin of `Jenkins`_, you can add another part::

    [buildout]
    parts =
        ...
        report-xml

    [report-xml]
    recipe = zc.recipe.egg
    eggs = coverage
    scripts = coverage=report-xml
    initialization =
        sys.argv = sys.argv[:] + ['xml', '-i']

This will generate a ``coverage.xml`` file in the buildout root.

Optional dependencies
---------------------

``plone.testing`` comes with a core set of tools for managing layers, which depends only on `zope.testing`_.
In addition, there are several layers and helper functions which can be used in your own tests (or as bases for your own layers).
Some of these have deeper dependencies.
However, these dependencies are optional and not installed by default.
If you don't use the relevant layers, you can safely ignore them.

``plone.testing`` does specify these dependencies, however, using the ``setuptools`` "extras" feature.
You can depend on one or more extras in your own ``setup.py`` ``install_requires`` or ``extras_require`` option using the same square bracket notation shown for the ``[test]`` buildout part above.
For example, if you need both the ``zca`` and ``publisher`` extras, you can have the following in your ``setup.py``::

    extras_require = {
        'test': [
                'plone.testing [zca, publisher]',
            ]
    },

The available extras are:

``zodb``
    ZODB testing.
    Depends on ``ZODB``.
    The relevant layers and helpers are in the module ``plone.testing.zodb``.

``zca``
    Zope Component Architecture testing.
    Depends on core Zope Component Architecture packages such as ``zope.component`` and ``zope.event``.
    The relevant layers and helpers are in the module ``plone.testing.zca``.

``security``
    Security testing.
    Depends on ``zope.security``.
    The relevant layers and helpers are in the module ``plone.testing.security``.

``publisher``
    Zope Publisher testing.
    Depends on ``zope.publisher``, ``zope.browsermenu``, ``zope.browserpage``, ``zope.browserresource`` and ``zope.security`` and sets up ZCML directives.
    The relevant layers and helpers are in the module ``plone.testing.publisher``.

``zope`` (For backwards compatibility there is also ``z2``.)

    Zope testing.
    Depends on the ``Zope`` egg, which includes all the dependencies of the Zope application server.
    The relevant layers and helpers are in the module ``plone.testing.zope``.

``zserver``

    Tests against the ``ZServer``. (Python 2 only!) Requires additionally to use the ``zope`` extra.
    The relevant layers and helpers are in the module ``plone.testing.zserver``


Adding a test buildout to your package
--------------------------------------

When creating reusable, mostly stand-alone packages, it is often useful to be able to include a buildout with the package sources itself that can be used to create a test runner.
This is a popular approach for many Zope packages, for example.
In fact, ``plone.testing`` itself uses this kind of layout.

To have a self-contained buildout in your package, the following is required:

* You need a ``buildout.cfg`` at the root of the package.

* In most cases, you always want a ``bootstrap.py`` file to make it easier for people to set up a fresh buildout.

* Your package sources need to be inside a ``src`` directory.
  If you're using namespace packages, that means the top level package should be in the ``src`` directory.

* The ``src`` directory must be referenced in ``setup.py``.

For example, ``plone.testing`` has the following layout::

    plone.testing/
    plone.testing/setup.py
    plone.testing/bootstrap.py
    plone.testing/buildout.cfg
    plone.testing/README.rst
    plone.testing/src/
    plone.testing/src/plone
    plone.testing/src/plone/__init__.py
    plone.testing/src/plone/testing/
    plone.testing/src/plone/testing/*

In ``setup.py``, the following arguments are required::

        packages=find_packages('src'),
        package_dir={'': 'src'},

This tells ``setuptools`` where to find the source code.

The ``buildout.cfg`` for ``plone.testing`` looks like this::

    [buildout]
    extends =
        http://download.zope.org/Zope2/index/2.12.12/versions.cfg
    parts = coverage test report report-xml
    develop = .

    [test]
    recipe = collective.xmltestreport
    eggs =
        plone.testing [test]
    defaults = ['--auto-color', '--auto-progress']

    [coverage]
    recipe = zc.recipe.egg
    eggs = coverage
    initialization =
        include = '--source=${buildout:directory}/src'
        sys.argv = sys.argv[:] + ['run', include, 'bin/test', '--all', '--xml']

    [report]
    recipe = zc.recipe.egg
    eggs = coverage
    scripts = coverage=report
    initialization =
        sys.argv = sys.argv[:] + ['html', '-i']

    [report-xml]
    recipe = zc.recipe.egg
    eggs = coverage
    scripts = coverage=report-xml
    initialization =
        sys.argv = sys.argv[:] + ['xml', '-i']

Obviously, you should adjust the package name in the ``eggs`` list and the version set in the ``extends`` line as appropriate.

You can of course also add additional buildout parts, for example to include some development/debugging tools, or even a running application server for testing purposes.

    *Hint:* If you use this package layout, you should avoid checking any files or directories generated by buildout into your version control repository.
    You want to ignore:

    * ``.coverage``
    * ``.installed.cfg``
    * ``bin``
    * ``coverage.xml``
    * ``develop-eggs``
    * ``htmlcov``
    * ``parts``
    * ``src/*.egg-info``

Layers
======

In large part, ``plone.testing`` is about layers.
It provides:

* A set of layers (outlined below), which you can use or extend.

* A set of tools for working with layers

* A mini-framework to make it easy to write layers and manage shared resources associated with layers.

We'll discuss the last two items here, before showing how to write tests that use layers.

Layer basics
------------

Layers are used to create test fixtures that are shared by multiple test cases.
For example, if you are writing a set of integration tests, you may need to set up a database and configure various components to access that database.
This type of test fixture setup can be resource-intensive and time-consuming.
If it is possible to only perform the setup and tear-down once for a set of tests without losing isolation between those tests, test runs can often be sped up significantly.

Layers also allow reuse of test fixtures and set-up/tear-down code.
``plone.testing`` provides a number of useful (but optional) layers that manage test fixtures for common Zope testing scenarios, letting you focus on the actual test authoring.

At the most basic, a layer is an object with the following methods and attributes:

``setUp()``
    Called by the test runner when the layer is to be set up.
    This is called exactly once for each layer used during a test run.

``tearDown()``
    Called by the test runner when the layer is to be torn down.
    As with ``setUp()``, this is called exactly once for each layer.

``testSetUp()``
    Called immediately before each test case that uses the layer is executed.
    This is useful for setting up aspects of the fixture that are managed on a per-test basis, as opposed to fixture shared among all tests.

``testTearDown()``
    Called immediately after each test case that uses the layer is executed.
    This is a chance to perform any post-test cleanup to ensure the fixture is ready for the next test.

``__bases__``
    A tuple of base layers.

Each test case is associated with zero or one layer.
(The syntax for specifying the layer is shown in the section "Writing tests" below.) All the tests associated with a given layer will be executed together.

Layers can depend on one another (as indicated in the ``__bases__`` tuple), allowing one layer to build on the fixture created by another.
Base layers are set up before and torn down after their dependants.

For example, if the test runner is executing some tests that belong to layer A, and some other tests that belong to layer B, both of which depend on layer C, the order of execution might be::

    1. C.setUp()
    1.1. A.setUp()

    1.1.1. C.testSetUp()
    1.1.2. A.testSetUp()
    1.1.3. [One test using layer A]
    1.1.4. A.testTearDown()
    1.1.5. C.testTearDown()

    1.1.6. C.testSetUp()
    1.1.7. A.testSetUp()
    1.1.8. [Another test using layer A]
    1.1.9. A.testTearDown()
    1.1.10. C.testTearDown()

    1.2. A.tearDown()
    1.3. B.setUp()

    1.3.1. C.testSetUp()
    1.3.2. B.testSetUp()
    1.3.3. [One test using layer B]
    1.3.4. B.testTearDown()
    1.3.5. C.testTearDown()

    1.3.6. C.testSetUp()
    1.3.7. B.testSetUp()
    1.3.8. [Another test using layer B]
    1.3.9. B.testTearDown()
    1.3.10. C.testTearDown()

    1.4. B.tearDown()
    2. C.tearDown()

A base layer may of course depend on other base layers.
In the case of nested dependencies like this, the order of set up and tear-down as calculated by the test runner is similar to the way in which Python searches for the method to invoke in the case of multiple inheritance.

Writing layers
--------------

The easiest way to create a new layer is to use the ``Layer`` base class and implement the ``setUp()``, ``tearDown()``, ``testSetUp()`` and ``testTearDown()`` methods as needed.
All four are optional.
The default implementation of each does nothing.

By convention, layers are created in a module called ``testing.py`` at the top level of your package.
The idea is that other packages that extend your package can reuse your layers for their own testing.

A simple layer may look like this::

    >>> from plone.testing import Layer
    >>> class SpaceShip(Layer):
    ...
    ...     def setUp(self):
    ...         print("Assembling space ship")
    ...
    ...     def tearDown(self):
    ...         print("Disasembling space ship")
    ...
    ...     def testSetUp(self):
    ...         print("Fuelling space ship in preparation for test")
    ...
    ...     def testTearDown(self):
    ...         print("Emptying the fuel tank")

Before this layer can be used, it must be instantiated.
Layers are normally instantiated exactly once, since by nature they are shared between tests.
This becomes important when you start to manage resources (such as persistent data, database connections, or other shared resources) in layers.

The layer instance is conventionally also found in ``testing.py``, just after the layer class definition.::

    >>> SPACE_SHIP = SpaceShip()

.. note::

    Since the layer is instantiated in module scope, it will be created as soon as the ``testing`` module is imported.
    It is therefore very important that the layer class is inexpensive and safe to create.
    In general, you should avoid doing anything non-trivial in the ``__init__()`` method of your layer class.
    All setup should happen in the ``setUp()`` method.
    If you *do* implement ``__init__()``, be sure to call the ``super`` version as well.

The layer shown above did not have any base layers (dependencies).
Here is an example of another layer that depends on it:::

    >>> class ZIGSpaceShip(Layer):
    ...     defaultBases = (SPACE_SHIP,)
    ...
    ...     def setUp(self):
    ...         print("Installing main canon")

    >>> ZIG = ZIGSpaceShip()

Here, we have explicitly listed the base layers on which ``ZIGSpaceShip`` depends, in the ``defaultBases`` attribute.
This is used by the ``Layer`` base class to set the layer bases in a way that can also be overridden: see below.

Note that we use the layer *instance* in the ``defaultBases`` tuple, not the class.
Layer dependencies always pertain to specific layer instances.
Above, we are really saying that *instances* of ``ZIGSpaceShip`` will, by default, require the ``SPACE_SHIP`` layer to be set up first.

.. note::

    You may find it useful to create other layer base/mix-in classes that extend ``plone.testing.Layer`` and provide helper methods for use in your own layers.
    This is perfectly acceptable, but please do not confuse a layer base class used in this manner with the concept of a *base layer* as described above:

    * A class deriving from ``plone.testing.Layer`` is known as a *layer class*.
      It defines the behaviour of the layer by implementing the lifecycle methods ``setUp()``, ``tearDown()``, ``testSetUp()`` and/or ``testTearDown()``.

    * A layer class can be instantiated into an actual layer.
      When a layer is associated with a test, it is the layer *instance* that is used.

    * The instance is usually a shared, module-global object, although in some cases it is useful to create copies of layers by instantiating the class more than once.

    * Subclassing an existing layer class is just straightforward OOP reuse: the test runner is not aware of the subclassing relationship.

    * A layer *instance* can be associated with any number of layer *bases*, via its ``__bases__`` property (which is usually via the ``defaultBases`` variable in the class body and/or overridden using the ``bases`` argument to the ``Layer`` constructor).
      These bases are layer *instances*, not classes.
      The test runner will inspect the ``__bases__`` attribute of each layer instance it sets up to calculate layer pre-requisites and dependencies.

    Also note that the `zope.testing`_ documentation contains examples of layers that are "old-style" classes where the ``setUp()`` and ``tearDown()`` methods are ``classmethod`` methods and class inheritance syntax is used to specify base layers.
    Whilst this pattern works, we discourage its use, because the classes created using this pattern are not really used as classes.
    The concept of layer bases is slightly different from class inheritance, and using the ``class`` keyword to create layers with base layers leads to a number of "gotchas" that are best avoided.

Advanced - overriding bases
---------------------------

In some cases, it may be useful to create a copy of a layer, but change its bases.
One reason to do this may if you are reusing a layer from another module, and you need to change the order in which layers are set up and torn down.

Normally, of course, you would just reuse the layer instance, either directly in a test, or in the ``defaultBases`` tuple of another layer, but if you need to change the bases, you can pass a new list of bases to the layer instance constructor:::

    >>> class CATSMessage(Layer):
    ...
    ...     def setUp(self):
    ...         print("All your base are belong to us")
    ...
    ...     def tearDown(self):
    ...         print("For great justice")

    >>> CATS_MESSAGE = CATSMessage()

    >>> ZERO_WING = ZIGSpaceShip(bases=(SPACE_SHIP, CATS_MESSAGE,), name="ZIGSpaceShip:CATSMessage")

Please note that when overriding bases like this, the ``name`` argument is required.
This is because each layer (using in a given test run) must have a unique name.
The default is to use the layer class name, but this obviously only works for one instantiation.
Therefore, ``plone.testing`` requires a name when setting ``bases`` explicitly.

Please take great care when changing layer bases like this.
The layer implementation may make assumptions about the test fixture that was set up by its bases.
If you change the order in which the bases are listed, or remove a base altogether, the layer may fail to set up correctly.

Also, bear in mind that the new layer instance is independent of the original layer instance, so any resources defined in the layer are likely to be duplicated.

Layer combinations
------------------

Sometimes, it is useful to be able to combine several layers into one, without adding any new fixture.
One way to do this is to use the ``Layer`` class directly and instantiate it with new bases:::

    >>> COMBI_LAYER = Layer(bases=(CATS_MESSAGE, SPACE_SHIP,), name="Combi")

Here, we have created a "no-op" layer with two bases: ``CATS_MESSAGE`` and ``SPACE_SHIP``, named ``Combi``.

Please note that when using ``Layer`` directly like this, the ``name`` argument is required.
This is to allow the test runner to identify the layer correctly.
Normally, the class name of the layer is used as a basis for the name, but when using the ``Layer`` base class directly, this is unlikely to be unique or descriptive.

Layer resources
---------------

Many layers will manage one or more resources that are used either by other layers, or by tests themselves.
Examples may include database connections, thread-local objects, or configuration data.

``plone.testing`` contains a simple resource storage abstraction that makes it easy to access resources from dependent layers or tests.
The resource storage uses dictionary notation:::

    >>> class WarpDrive(object):
    ...     """A shared resource"""
    ...
    ...     def __init__(self, maxSpeed):
    ...         self.maxSpeed = maxSpeed
    ...         self.running = False
    ...
    ...     def start(self, speed):
    ...         if speed > self.maxSpeed:
    ...             print("We need more power!")
    ...         else:
    ...             print("Going to warp at speed", speed)
    ...             self.running = True
    ...
    ...     def stop(self):
    ...         self.running = False

    >>> class ConstitutionClassSpaceShip(Layer):
    ...     defaultBases = (SPACE_SHIP,)
    ...
    ...     def setUp(self):
    ...         self['warpDrive'] = WarpDrive(8.0)
    ...
    ...     def tearDown(self):
    ...         del self['warpDrive']

    >>> CONSTITUTION_CLASS_SPACE_SHIP = ConstitutionClassSpaceShip()

    >>> class GalaxyClassSpaceShip(Layer):
    ...     defaultBases = (CONSTITUTION_CLASS_SPACE_SHIP,)
    ...
    ...     def setUp(self):
    ...         # Upgrade the warp drive
    ...         self.previousMaxSpeed = self['warpDrive'].maxSpeed
    ...         self['warpDrive'].maxSpeed = 9.5
    ...
    ...     def tearDown(self):
    ...         # Restore warp drive to its previous speed
    ...         self['warpDrive'].maxSpeed = self.previousMaxSpeed

    >>> GALAXY_CLASS_SPACE_SHIP = GalaxyClassSpaceShip()

As shown, layers (that derive from ``plone.testing.Layer``) support item (dict-like) assignment, access and deletion of arbitrary resources under string keys.

    **Important:** If a layer creates a resource (by assigning an object to a key on ``self`` as shown above) during fixture setup-up, it must also delete the resource on tear-down.
    Set-up and deletion should be symmetric: if the resource is assigned during ``setUp()`` it should be deleted in ``tearDown()``;
    if it's created in ``testSetUp()`` it should be deleted in ``testTearDown()``.

A resource defined in a base layer is accessible from and through a child layer.
If a resource is set on a child using a key that also exists in a base layer, the child version will shadow the base version until the child layer is torn down (presuming it deletes the resource, which it should), but the base layer version remains intact.

.. note::

    Accessing a resource is analogous to accessing an instance variable.
    For example, if a base layer assigns a resource to a given key in its ``setUp()`` method, a child layer shadows that resource with another object under the same key, the shadowed resource will by used during the ``testSetUp()`` and ``testTearDown()`` lifecycle methods if implemented by the *base* layer as well.
    This will be the case until the child layer "pops" the resource by deleting it, normally in its ``tearDown()``.

Conversely, if (as shown above) the child layer accesses and modifies the object, it will modify the original.

.. note::

   It is sometimes necessary (or desirable) to modify a shared resource in a child layer, as shown in the example above.  In this case, however, it is very important to restore the original state when the layer is torn down.  Otherwise, other layers or tests using the base layer directly may be affected in difficult-to-debug ways.

If the same key is used in multiple base layers, the rules for choosing which version to use are similar to those that apply when choosing an attribute or method to use in the case of multiple inheritance.

In the example above, we used the resource manager for the ``warpDrive`` object, but we assigned the ``previousMaxSpeed`` variable to ``self``.
This is because ``previousMaxSpeed`` is internal to the layer and should not be shared with any other layers that happen to use this layer as a base.
Nor should it be used by any test cases.
Conversely, ``warpDrive`` is a shared resource that is exposed to other layers and test cases.

The distinction becomes even more important when you consider how a test case may access the shared resource.
We'll discuss how to write test cases that use layers shortly, but consider the following test:::

    >>> import unittest
    >>> class TestFasterThanLightTravel(unittest.TestCase):
    ...     layer = GALAXY_CLASS_SPACE_SHIP
    ...
    ...     def test_hyperdrive(self):
    ...         warpDrive = self.layer['warpDrive']
    ...         warpDrive.start(8)

This test needs access to the shared resource.
It knows that its layer defines one called ``warpDrive``.
It does not know or care that the warp drive was actually initiated by the ``ConstitutionClassSpaceShip`` base layer.

If, however, the base layer had assigned the resource as an instance variable, it would not inherit to child layers (remember: layer bases are not base classes!).
The syntax to access it would be:::

    self.layer.__bases__[0].warpDrive

which is not only ugly, but brittle: if the list of bases is changed, the expression above may lead to an attribute error.

Writing tests
=============

Tests are usually written in one of two ways: As methods on a class that derives from ``unittest.TestCase`` (this is sometimes known as "Python tests" or "JUnit-style tests"), or using doctest syntax.

You should realise that although the relevant frameworks (``unittest`` and ``doctest``) often talk about unit testing, these tools are also used to write integration and functional tests.
The distinction between unit, integration and functional tests is largely practical: you use the same techniques to set up a fixture or write assertions for an integration test as you would for a unit test.
The difference lies in what that fixture contains, and how you invoke the code under test.
In general, a true unit test will have a minimal or no test fixture, whereas an integration test will have a fixture that contains the components your code is integrating with.
A functional test will have a fixture that contains enough of the full system to execute and test an "end-to-end" scenario.

Python tests
------------

Python tests use the Python `unittest`_ module.
They should be placed in a module or package called ``tests`` for the test runner to pick them up.

For small packages, a single module called ``tests.py`` will normally contain all tests.
For larger packages, it is common to have a ``tests`` package that contains a number of modules with tests.
These need to start with the word ``test``, e.g.
``tests/test_foo.py`` or ``tests/test_bar.py``.
Don't forget the ``__init__.py`` in the ``tests`` package, too!

unittest
~~~~~~~~

Please note that the `zope.testing`_ test runner at the time of writing (version 4.6.2) does not (yet) support the new ``setUpClass()``, ``tearDownClass()``, ``setUpModule()`` and ``tearDownModule()`` hooks from ``unittest``.
This is not normally a problem, since we tend to use layers to manage complex fixtures, but it is important to be aware of nonetheless.

Test modules, classes and functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Python tests are written with classes that derive from the base class ``TestCase``.
Each test is written as a method that takes no arguments and has a name starting with ``test``.
Other methods can be added and called from test methods as appropriate, e.g.
to share some test logic.

Two special methods, ``setUp()`` and ``tearDown()``, can also be added.
These will be called before or after each test, respectively, and provide a useful place to construct and clean up test fixtures without writing a custom layer.
They are obviously not as reusable as layers, though.

   *Hint:* Somewhat confusingly, the ``setUp()`` and ``tearDown()`` methods in a test case class are the equivalent of the ``testSetUp()`` and ``testTearDown()`` methods of a layer class.

A layer can be specified by setting the ``layer`` class attribute to a layer instance.
If layers are used in conjunction with ``setUp()`` and ``tearDown()`` methods in the test class itself, the class' ``setUp()`` method will be called after the layer's ``testSetUp()`` method, and the class' ``tearDown()`` method will be called before the layer's ``testTearDown()`` method.

The ``TestCase`` base class contains a number of methods which can be used to write assertions.
They all take the form ``self.assertSomething()``, e.g.
``self.assertEqual(result, expectedValue)``.
See the `unittest`_ documentation for details.

Putting this together, let's expand on our previous example unit test:::

    >>> import unittest

    >>> class TestFasterThanLightTravel(unittest.TestCase):
    ...     layer = GALAXY_CLASS_SPACE_SHIP
    ...
    ...     def setUp(self):
    ...         self.warpDrive = self.layer['warpDrive']
    ...         self.warpDrive.stop()
    ...
    ...     def tearDown(self):
    ...         self.warpDrive.stop()
    ...
    ...     def test_warp8(self):
    ...         self.warpDrive.start(8)
    ...         self.assertEqual(self.warpDrive.running, True)
    ...
    ...     def test_max_speed(self):
    ...         tooFast = self.warpDrive.maxSpeed + 0.1
    ...         self.warpDrive.start(tooFast)
    ...         self.assertEqual(self.warpDrive.running, False)

A few things to note:

* The class derives from ``unittest.TestCase``.

* The ``layer`` class attribute is set to a layer instance (not a layer class!) defined previously.
  This would typically be imported from a ``testing`` module.

* There are two tests here: ``test_warp8()`` and ``test_max_speed()``.

* We have used the ``self.assertEqual()`` assertion in both tests to check the result of executing the ``start()`` method on the warp drive.

* We have used the ``setUp()`` method to fetch the ``warpDrive`` resource and ensure that it is stopped before each test is executed.
  Assigning a variable to ``self`` is a useful way to provide some state to each test method, though be careful about data leaking between tests: in general, you cannot predict the order in which tests will run, and tests should always be independent.

* We have used the ``tearDown()`` method to make sure the warp drive is really stopped after each test.

Test suites
~~~~~~~~~~~

A class like the one above is all you need: any class deriving from ``TestCase`` in a module with a name starting with ``test`` will be examined for test methods.
Those tests are then collected into a test suite and executed.

See the `unittest`_ documentation for other options.

Doctests
--------

Doctests can be written in two ways: as the contents of a docstring (usually, but not always, as a means of illustrating and testing the functionality of the method or class where the docstring appears), or as a separate text file.
In both cases, the standard `doctest`_ module is used.
See its documentation for details about doctest syntax and conventions.

Doctests are used in two different ways:

* To test documentation.
  That is, to ensure that code examples contained in documentation are valid and continue to work as the software is updated.

* As a convenient syntax for writing tests.

These two approaches use the same testing APIs and techniques.
The difference is mostly about mindset.
However, it is important to avoid falling into the trap that tests can substitute for good documentation or vice-a-versa.
Tests usually need to systematically go through inputs and outputs and cover off a number of corner cases.
Documentation should tell a compelling narrative and usually focus on the main usage scenarios.
Trying to kill these two birds with one stone normally leaves you with an unappealing pile of stones and feathers.

Docstring doctests
~~~~~~~~~~~~~~~~~~

Doctests can be added to any module, class or function docstring:::

    def canOutrunKlingons(warpDrive):
        """Find out of the given warp drive can outrun Klingons.

        Klingons travel at warp 8

        >>> drive = WarpDrive(5)
        >>> canOutrunKlingons(drive)
        False

        We have to be faster than that to outrun them.

        >>> drive = WarpDrive(8.1)
        >>> canOutrunKlingons(drive)
        True

        We can't outrun them if we're travelling exactly the same speed

        >>> drive = WarpDrive(8.0)
        >>> canOutrunKlingons(drive)
        False

        """
        return warpDrive.maxSpeed > 8.0

To add the doctests from a particular module to a test suite, you need to use the ``test_suite()`` function hook:::

    >>> import doctest
    >>> def test_suite():
    ...     suite = unittest.TestSuite()
    ...     suite.addTests([
    ...         unittest.defaultTestLoader.loadTestsFromTestCase(TestFasterThanLightTravel), # our previous test
    ...         doctest.DocTestSuite('spaceship.utils'),
    ...     ])
    ...     return suite

Here, we have given the name of the module to check as a string dotted name.
It is also possible to import a module and pass it as an object.
The code a