zeit.nightwatch 2.3.2
pytest helpers for http smoke tests

.. image:: https://github.com/ZeitOnline/zeit.nightwatch/workflows/Run%20tests/badge.svg
   :alt: Test status badge


===============
zeit.nightwatch
===============

pytest helpers for http smoke tests


Making HTTP requests
====================

``zeit.nightwatch.Browser`` wraps a `requests <https://pypi.org/project/requests/>`_ ``Session`` to provide some convenience features:

- Instantiate with a base url, and then only use paths:
  ``http = Browser('https://example.com'); http.get('/foo')``
  will request https://example.com/foo
- A convenience ``http`` fixture is provided, which can be configured via the ``nightwatch_config`` fixture.
- Use call instead of get, because it's just that *little bit* shorter.
  (``http('/foo')`` instead of ``http.get('/foo')``)
- Fill and submit forms, powered by `mechanicalsoup <https://pypi.org/project/MechanicalSoup/>`_.
  (We've customized this a bit, so that responses are only parsed with beautifulsoup if a feature like forms or links is actually used.)
- Logs request and response headers, so pytest prints these on test failures, to help debugging.
- Use ``sso_login(username, password)`` to log into https://meine.zeit.de.
- See source code for specific API details.


Example usage::

    @pytest.fixture(scope='session')
    def nightwatch_config():
        return dict(browser=dict(
            baseurl='https://example.com',
            sso_url='https://meine.zeit.de/anmelden',
        ))

    def test_my_site(http):
        r = http.get('/something')
        assert r.status_code == 200

    def test_login(http):
        http('/login')
        http.select_form()
        http.form['username'] = 'joe@example.com'
        http.form['password'] = 'secret'
        r = http.submit()
        assert '/home' in r.url

    def test_meinezeit_redirects_to_konto_after_login(http):
        r = http.sso_login('joe@example.com', 'secret')
        assert r.url == 'https://www.zeit.de/konto'


Examining HTML responses
========================

nightwatch adds two helper methods to the ``requests.Response`` object:

* ``xpath()``: parses the response with ``lxml.html`` and then calls ``xpath()`` on that document
* ``css()``: converts the selector to xpath using `cssselect <https://pypi.org/project/cssselect/>`_ and then calls ``xpath()``


Example usage::

    def test_error_page_contains_home_link(http):
        r = http('/nonexistent')
        assert r.status_code == 404
        assert r.css('a.home')


Controlling a browser with playwright
=====================================

nightwatch pulls in the ``pytest-playwright`` plugin, so you can use their fixtures.

Unfortunately, the playwright API is too unfriendly to allow nightwatch to set the base url automatically,
so you'll need to do that yourself, for example by overriding the ``base_url`` fixture::

    @pytest.fixture(scope="session")
    def base_url():
        return 'https://example.com'

    def test_playwright_works(page):
        page.goto('/something')


Running against different environments
======================================

To help with running the same tests against e.g. a staging and production environment, nightwatch declares a pytest commandline option ``--nightwatch-environment``.

A pattern we found helpful is using a fixture to provide environment-specific settings, like this::

    CONFIG_STAGING = {
        'base_url': 'https://staging.example.com',
        'username': 'staging_user',
        'password': 'secret',
    }

    CONFIG_PRODUCTION = {
        'base_url': 'https://www.example.com',
        'username': 'production_user',
        'password': 'secret2',
    }

    @pytest.fixture(scope='session')
    def nightwatch_config(nightwatch_environment):
        config = globals()['CONFIG_%s' % nightwatch_environment.upper()]
        return dict(environment=nightwatch_environment, browser=config)

    def test_some_integration_that_has_no_staging(http, nightwatch_config):
        if nightwatch_config['environment'] != 'production':
            pytest.skip('The xyz integration has no staging')
        r = http('/trigger-xyz')
        assert r.json()['message'] == 'OK'


Sending test results to prometheus
==================================

Like the medieval night watch people who made the rounds checking that doors were locked,
our use case for this library is continuous black box high-level tests that check that main functional areas of our systems are working.

For this purpose, we want to integrate the test results with our monitoring system, which is based on `Prometheus <https://prometheus.io>`_.
We've taken inspiration from the `pytest-prometheus <https://pypi.org/project/pytest-prometheus/>`_ plugin, and tweaked it a little to use a stable metric name, so we can write a generic alerting rule.

This uses the configured `Pushgateway <https://prometheus.io/docs/practices/pushing/>`_ to record metrics like this::

    nightwatch_check{test="test_error_page_contains_home_link",environment="staging",project="website",job="website-staging"}=1  # pass=1, fail=0

The ``environment`` label is populated from ``--nightwatch-environment``, see above,
and the ``project`` label is populated from an environment variable ``NIGHTWATCH_NAMESPACE`` if present
(this can be set e.g. via `k8s Downward API <https://kubernetes.io/docs/concepts/workloads/pods/downward-api/#available-fields>`_).
(Note that we use a separate ``project`` label, since the ``namespace`` label is occupied by the pushgateway itself and thus does not help.)

This functionality is disabled by default, nightwatch declares a pytest commandline option ``--prometheus`` which has to be present to enable pushing the metrics.
There also are commandline options to override the pushgateway url etc., please see the source code for those details.


Sending test results to elasticsearch
=====================================

We're running our tests as kubernetes pods, and their stdout/stderr output is captured and sent to elasticsearch.
However the normal pytest output is meant for humans, but is not machine-readable.
Thus we've implemented a JSON lines test report format that can be enabled with ``--json-report=filename`` or ``--json-report=-`` to directly send to stdout.

Here's an output example, formatted for readability (in reality, each test produces a single JSON line, since that's what our k8s log processor expects)::

    {
      "time": "2023-12-08T10:37:40.630617+00:00",
      "test_stage": "call",
      "test_class": "smoketest.test_api",
      "test_name": "test_example",
      "test_outcome": "passed",
      "system_log": "11:37:40 INFO  [zeit.nightwatch.requests][MainThread] > POST http://example.com/something\n..."
    }


Development and testing
=======================

Pull requests will run tests in workflow via `bin/test` and do test_basics.py. The two tests from test_sso_login.py will be skipped. These two tests will be only done locally if you have set your vault token and testing credentials are setup.


zeit.nightwatch changes
=======================

.. towncrier release notes start

2.3.2 (2025-04-11)
------------------

Changes
+++++++

- Correctly determine prometheus job name from environment variables (prometheus)


2.3.1 (2025-04-08)
------------------

Changes
+++++++

- Also wait for return_url redirect after playwright sso login (ssowait)


2.3.0 (2025-04-03)
------------------

Changes
+++++++

- Wait for /konto redirect after playwright sso login (ssowait)


2.2.0 (2024-12-13)
------------------

Changes
+++++++

- Record prometheus metric on setup/teardown failure (prometheus)


2.1.0 (2024-12-13)
------------------

Changes
+++++++

- Set project label and job name automatically from k8s namespace environment variable (prometheus)


2.0.0 (2024-12-13)
------------------

Changes
+++++++

- Breaking change: Projects have to set base_url via a fixture now, setting nightwatch_config["selenium"] was removed due to unreliability (playwright)
- Remove selenium support (selenium)


1.11.0 (unreleased)
-------------------

- Nothing changed yet.


1.10.0 (2024-12-13)
-------------------

- Add our own `gcs-upload` helper script, since gsutil does not support gcloud auth anymore


1.9.2 (2024-12-12)
------------------

- Add infrastructure support for generating HTML reports and upload them to GCS


1.9.1 (2024-06-04)
------------------

- Apply convenience playwright Page.sso_login monkeypatch at import-time


1.9.0 (2024-06-04)
------------------

- Support keycloak for SSO


1.8.0 (2024-05-13)
------------------

- Also log HTTP response body, not just headers.


1.7.1 (2023-12-13)
------------------

- Don't try to json report if no argument was given


1.7.0 (2023-12-08)
------------------

- Implement ``--json-report=-`` for line-based output


1.6.0 (2022-12-16)
------------------

- Support playwright


1.5.1 (2022-06-24)
------------------

- Use non-deprecated selenium API


1.5.0 (2022-03-25)
------------------

- Support `selenium-wire` in addition to `selenium`


1.4.2 (2022-02-21)
------------------

- ZO-712: Set referer explicitly during sso_login, required for csrf validation


1.4.1 (2021-10-27)
------------------

- Include tests & setup in tarball to support `devpi test`


1.4.0 (2021-10-26)
------------------

- Add patch to requests


1.3.3 (2021-04-01)
------------------

- Support contains instead of equals for `find_link`


1.3.2 (2021-02-18)
------------------

- Record skipped tests as passed to prometheus, not failed


1.3.1 (2021-02-17)
------------------

- Handle same metric name (and testname only as label) correctly


1.3.0 (2021-02-17)
------------------

- Allow to configure the test browsers via a config fixture


1.2.0 (2021-02-17)
------------------

- Add convenience `nightwatch` fixture and toplevel API

- Add first test & fix package setup


1.1.0 (2021-02-12)
------------------

- Include prometheus functionality here, to fix pushgateway bug
  and support sending the test name as a label.

- Declare namespace package properly


1.0.0 (2021-02-11)
------------------

- Initial release