omfit-testing 2020.12.2.22.59
omfit_testing
Requires Python: >=3.6

**TESTING** files used as part of the OMFIT project: https://omfit.io/

Provides:
=========
* OMFITtest
* manage_tests
* setup_warnings
* get_server_name
* clear_old_test_report_comments
* auto_comment_marker
* run_test_outside_framework
* RedirectStdStreams
* auto_test_mode
* standard_test_keywords

Requires:
=========
* matplotlib>=3.1,!=3.2.1,!=3.2.2
* numpy>=1.12
* pyyaml>=3.13
* requests>=2.20
* uncertainties>=3
* omfit_github==2020.12.2.22.59
* omfit_commonclasses==2020.12.2.22.59

Authors:
=========
https://omfit.io/contributors.html

Documentation:
==============
OMFITtest
---------

    Test case with some extra methods to help with OMFIT testing tasks

    To use this class, make your own class that is a subclass of this one:
    >> class TestMyStuff(OMFITtest):
    >>     notify_gh_status = True  # Example of a test setting you can override
    In the top of your file, override key test settings as needed

    Test settings you can override by defining them at the top of your class:
    -------------------------------
    :param warning_level: int
        Instructions for turning some warnings into exceptions & ignoring others
        -1: Make no changes to warnings
        0: No exceptions from warnings
        1: Ignores some math warnings related to NaNs & some which should be
            benign. Exceptions for other warnings.
        2: (RECOMMENDED) Ignores a small set of warnings which are probably
            benign, but exceptions for everything else.
        3: Exceptions for practically every warning. Only ignores some really
            inconsequential ones from OMFIT.
        4: No warnings allowed. Always throw exceptions!
        The warnings are changed before your test starts, so you can still
        override or change them in s() or __init__().

    :param count_figs: bool
        Enable counting of figures. Manage using collect_figs(n) after opening
        n figures. The actual figure count will be compared to the expected
        count (supplied by you as the argument), resulting in an AssertionError
        if the count does not match.

    :param count_guis: bool
        Enable counting of GUIs. Manage using collect_guis(n) after opening
        n GUIs. AssertionError if GUI count does not match expectation given
        via argument.

    :param leave_figs_open: bool
        Don't close figures at the end of each test (can lead to clutter)

    :param modules_to_load: list of strings or tuples
        Orders OMFIT to load the modules as indicated.
        Strings: modules ID. Tuples: (module ID, key)

    :param report_table: bool
        Keep a table of test results to include in final report

    :param table_sorting_columns: list of strings
        Names of columns to use for sorting table. Passed to table's group_by().
        This is most useful if you are adding extra columns to the results table
        during your test class's __init__() and overriding tearDown() to
        populate them.

    :param notify_gh_comment: int
        Turn on automatic report in a GitHub comment
        0: off
        1: always try to post or edit
        2: try to post or edit on failure only
        4: edit a comment instead of posting a new one if possible; only post if necessary
        8: edit the top comments and append test report or replace existing report with same context
        5 = behaviors of 4 and 1 (for example)

    :param notify_gh_status: bool
        Turn on automatic GitHub commit status updates

    :param gh_individual_status: int
        0 or False: No individual status contexts.
            Maximum of one status report if notify_gh_status is set.
        1 or True: Every test gets its own status context, including a pending
            status report when the test starts.
        2: Tests get their own status context only if they fail.
            No pending status for individual tests.
        3: Like 2 but ignores notify_gh_status and posts failed status reports
            even if reports are otherwise disabled.

    :param gh_overall_status_after_individual: bool
        Post the overall status even if individual status reports are enabled
        (set to 1 or True). Otherwise, the individual contexts replace the
        overall context. The overall status will be posted if individual status
        reports are set to 2 (only on failure).

    :param notify_slack: bool
        Post automatic test report on slack

    :param notify_email: bool
        Automatically send a test report to the user via email

    :param slack_channel: string
        Slack's code for the channel where the slack report should be posted,
        like "CHLS03MQC".

    :param slack_username: string
        The name to use for the apparent author of the slack post

    :param save_stats_to: dict-like
        Container for catching test statistics

    :param stats_key: string [optional]
        Test statistics are saved to save_stats_to[stats_key].
        stats_key will be automatically generated using the subclass name and a
        timestamp if left as None.

    :param topics_skipped: list of strings
        Provide a list of skipped test topics in order to have them included in
        the test report. The logic for skipping probably happens in the setup of
        your subclass, so we can't easily generate this list here.

    :param omfitx: reference to OMFITx
        Provide a reference to OMFITx to enable GUI counting and closing.
        This class might not be able to find OMFITx by itself, depending on how
        it is loaded.
    

manage_tests
------------

    Utility for running a set of OMFITtest-based test suites
    Example usage:
    >> class TestOmfitThingy(OMFITtest):
    >>     def test_thingy_init(self):
    >>         assert 1 != 0, '1 should not be 0'
    >> manage_tests(TestOmfitThingy)

    :param tests: OMFITtest instance or list of OMFITtest instances
        Define tests to run

    :param failfast: bool
        Passed straight to unittest. Causes execution to stop at the first error
        instead of running all tests and reporting which pass/fail.

    :param separate: bool
        Run each test suite separately and give it a separate context. Otherwise
        they'll have a single combined context.

    :param combined_context_name: string [optional]
        If not separate, override the automatic name
        ('+'.join([test.__class__.__name__) for test in tests]))

    :param force_gh_status: bool [optional]
        If None, use GitHub status post settings from the items in tests.
        If True or False: force status updates on/off.

    :param force_gh_comment: bool or int [optional]
        Like force_gh_status, but for comments, and with extra options:
        Set to 2 to post comments only on failure.

    :param force_email: bool [optional]
        None: notify_email on/off defined by test.
        True or False: force email notifications to be on or off.

    :param force_slack: bool [optional]
        None: notify_slack on/off defined by test.
        True or False: force slack notifications to be on or off.

    :param force_warning_level: int [optional]
        None: warning_level defined by test.
        int: Force the warning level for all tests to be this value.

    :param print_report: bool
        Print to console / command line?

    :param there_can_be_only_one: True, None or castable as int
        This value is interpreted as a set of binary flags. So 6 should be interpreted as options 2 and 4 active.

        A value of True is converted into 255 (all the bits are True, including unused bits).
        A value of None is replaced by the default value, which is True or 255.
        A float or string will work if it can be converted safely by int().

        1: Any of the flags will activate this feature. The 1 bit has no special meaning beyond activation.
            If active, old github comments will be deleted. The newest report may be retained.
        2: Limit deletion to reports that match the combined context of the test being run.
        4: Only protect the latest comment if it reports a failure; if the last test passed, all comments may be deleted
        8: Limit scope to comments with matching username

    :param raise_if_errors: bool
        Raise an OMFITexception at the end if there were any errors

    :param max_table_width: int
        Width in columns for the results table, if applicable. If too small,
        some columns will be hidden. Set to -1 to allow the table to be any
        width.

    :param set_gh_status_keywords: dict [optional]
        Dictionary of keywords to pass to set_gh_status()

    :param post_comment_to_github_keywords: dict [optional]
        Dictionary of keywords to pass to post_comment_to_github(), like thread, org, repository, and token

    :param ut_verbosity: int
        Verbosity level for unittest. 1 is normal, 0 suppresses ., E, and F reports from unittest as it runs.

    :param ut_stream:
        Output stream for unittest, such as StingIO() or sys.stdout

    :param only_these: string or list of strings
        Names of test units to run (with or without leading `test_`). Other test units will not be run. (None to run all tests)

    :param kw: quietly ignores other keywords

    :return: tuple containing:
        list of unittest results
        astropy.table.Table instance containing test results
        string reporting test results, including a formatted version of the table
    

setup_warnings
--------------

    A context manager like `catch_warnings`, that copies and restores the warnings
    filter upon exiting the context, with preset levels of warnings that turn some
    warnings into exceptions.

    :param record: specifies whether warnings should be captured by a
        custom implementation of warnings.showwarning() and be appended to a list
        returned by the context manager. Otherwise None is returned by the context
        manager. The objects appended to the list are arguments whose attributes
        mirror the arguments to showwarning().

    :param module: to specify an alternative module to the module
        named 'warnings' and imported under that name. This argument is only useful
        when testing the warnings module itself.

    :param level: (int) Controls how many warnings should throw errors

        -1: Do nothing at all and return immediately

        0: No warnings are promoted to exceptions. Specific warnings defined in
            higher levels are ignored and the rest appear as warnings, but with
            'always' instead of 'default' behavior: they won't disappear after
            the first instance.

        All higher warning levels turn all warnings into exceptions and then
        selectively ignore some of them:

        1: Ignores everything listed in level 2, but also ignores many common
            math errors that produce NaN.

        2: RECOMMENDED: In addition to level 3, also ignores several warnings
            of low-importance, but still leaves many math warnings (divide by 0)
            as errors.

        3: Ignores warnings which are truly irrelevant to almost any normal
            regression testing, such as the warning about not being able to make
            backup copies of scripts that are loaded in developer mode. Should
            be about as brutal as level 4 during the actual tests, but somewhat
            more convenient while debugging afterward.

        4: No warnings are ignored. This will be really annoying and not useful
            for many OMFIT applications.
    

get_server_name
---------------

    Returns the hostname.
    For known servers, it sanitizes hostname; e.g. irisd.cluster --> iris.gat.com
    

clear_old_test_report_comments
------------------------------

    Removes old automatic test reports
    :param lvl: int
        Interpreted as a set of binary flags, so 7 means options 1, 2, and 4 are active.
        1: Actually execute the deletion commands instead of just testing. In test mode, it returns list of dicts
           containing information about comments that would be deleted.
        2: Limit scope to current context (must supply contexts) and do not delete automatic comments from other context
        4: Do not preserve the most recent report unless it describes a failure
        8: Only target comments with matching username
    :param keyword: string [optional]
        The marker for deletion. Comments containing this string are gathered. The one with the latest timestamp
        is removed from the list. The rest are deleted. Defaults to the standard string used to mark automatic comments.
    :param contexts: string or list of strings [optional]
        Context(s) for tests to consider. Relevant only when scope is limited to present context.
    :param remove_all: bool
        Special case: don't exclude the latest comment from deletion because its status was already resolved. This comes
        up when the test would've posted a comment and then immediately deleted it and just skips posting. In this case,
        the actual last comment is not really the last comment that would've existed had we not skipped posting, so
        don't protect it.
    :param \**kw: optional keywords passed to delete_matching_gh_comments:
        thread: int [optional]
            Thread#, like pull request or issue number.
            Will be looked up automatically if you supply None and the current branch has an open pull request.
        token: string [optional]
            Token for accessing Github.
            Will be defined automatically if you supply None and you
            have previously stored a token using set_OMFIT_GitHub_token()
        org: string [optional]
            Organization on github, like `'gafusion'`.
            Will be looked up automatically if you supply None and the current branch has an open pull request.
        repository: string [optional]
            Repository on github within the org
            Will be looked up automatically if you supply None and the current branch has an open pull request.
    :return: list
        List of responses from github API requests, one entry per comment that the function attempts to delete.
        If the test keyword is set, this will be converted into a list of dicts with information about the comments that
        would be targeted.
    

auto_comment_marker
-------------------
str(object='') -> str
str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or
errors is specified, then the object must expose a data buffer
that will be decoded using the given encoding and error handler.
Otherwise, returns the result of object.__str__() (if defined)
or repr(object).
encoding defaults to sys.getdefaultencoding().
errors defaults to 'strict'.

run_test_outside_framework
--------------------------

    Deploys a test script outside the framework. Imports will be different.

    To include this in a test unit, do
    >>> return_code, log_tail = run_test_outside_framework(__file__)

    :param test_script: string
        Path to the file you want to test.
        If a test has a unit to test itself outside the framework, then this should be __file__.
        Also make sure a test can't run itself this way if it's already outside the framework.

    :param catch_exp_reports: bool
        Try to grab the end of the log starting with exception reports.
        Only works if test_script merges stdout and stderr; otherwise the exception reports will be somewhere else.
        You can use `with RedirectedStdStreams(stderr=sys.stdout):` in your code to do the merging.

    :return: (int, string)
        Return code (0 is success)
        End of output
    

RedirectStdStreams
------------------
Redirects stdout and stderr streams so you can merge them and get an easier to read log from a test.

auto_test_mode
--------------
bool(x) -> bool

Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.

standard_test_keywords
----------------------
dict() -> new empty dictionary
dict(mapping) -> new dictionary initialized from a mapping object's
    (key, value) pairs
dict(iterable) -> new dictionary initialized as if via:
    d = {}
    for k, v in iterable:
        d[k] = v
dict(**kwargs) -> new dictionary initialized with the name=value pairs
    in the keyword argument list.  For example:  dict(one=1, two=2)