jars/junit5-r5.10.2/documentation/src/docs/asciidoc/user-guide/advanced-topics/testkit.adoc

[[testkit]]
=== JUnit Platform Test Kit

The `junit-platform-testkit` artifact provides support for executing a test plan on the
JUnit Platform and then verifying the expected results. As of JUnit Platform 1.4, this
support is limited to the execution of a single `TestEngine` (see <<testkit-engine>>).

[[testkit-engine]]
==== Engine Test Kit

The `{testkit-engine-package}` package provides support for executing a `{TestPlan}` for a
given `{TestEngine}` running on the JUnit Platform and then accessing the results via a
fluent API to verify the expected results. The key entry point into this API is the
`{EngineTestKit}` which provides static factory methods named `engine()` and `execute()`.
It is recommended that you select one of the `engine()` variants to benefit from the
fluent API for building a `LauncherDiscoveryRequest`.

NOTE: If you prefer to use the `LauncherDiscoveryRequestBuilder` from the `Launcher` API
to build your `LauncherDiscoveryRequest`, you must use one of the `execute()` variants in
`EngineTestKit`.

The following test class written using JUnit Jupiter will be used in subsequent examples.

[[testkit-engine-ExampleTestCase]]
[source,java,indent=0]
----
include::{testDir}/example/ExampleTestCase.java[tags=user_guide]
----

For the sake of brevity, the following sections demonstrate how to test JUnit's own
`JupiterTestEngine` whose unique engine ID is `"junit-jupiter"`. If you want to test your
own `TestEngine` implementation, you need to use its unique engine ID. Alternatively, you
may test your own `TestEngine` by supplying an instance of it to the
`EngineTestKit.engine(TestEngine)` static factory method.

[[testkit-engine-statistics]]
==== Asserting Statistics

One of the most common features of the Test Kit is the ability to assert statistics
against events fired during the execution of a `TestPlan`. The following tests demonstrate
how to assert statistics for _containers_ and _tests_ in the JUnit Jupiter `TestEngine`.
For details on what statistics are available, consult the Javadoc for `{EventStatistics}`.

[source,java,indent=0]
----
include::{testDir}/example/testkit/EngineTestKitStatisticsDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<testkit-engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _container_ events.
<5> Assert statistics for _container_ events.
<6> Filter by _test_ events.
<7> Assert statistics for _test_ events.

NOTE: In the `verifyJupiterContainerStats()` test method, the counts for the `started` and
`succeeded` statistics are `2` since the `JupiterTestEngine` and the
<<testkit-engine-ExampleTestCase, `ExampleTestCase`>> class are both considered containers.

[[testkit-engine-events]]
==== Asserting Events

If you find that <<testkit-engine-statistics, asserting statistics>> alone is insufficient
for verifying the expected behavior of test execution, you can work directly with the
recorded `{Event}` elements and perform assertions against them.

For example, if you want to verify the reason that the `skippedTest()` method in
<<testkit-engine-ExampleTestCase, `ExampleTestCase`>> was skipped, you can do that as
follows.

[TIP]
====
The `assertThatEvents()` method in the following example is a shortcut for
`org.assertj.core.api.Assertions.assertThat(events.list())` from the {AssertJ} assertion
library.

For details on what _conditions_ are available for use with AssertJ assertions against
events, consult the Javadoc for `{EventConditions}`.
====

[source,java,indent=0]
----
include::{testDir}/example/testkit/EngineTestKitSkippedMethodDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the `skippedTest()` method in the <<testkit-engine-ExampleTestCase,
    `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _test_ events.
<5> Save the _test_ `Events` to a local variable.
<6> Optionally assert the expected statistics.
<7> Assert that the recorded _test_ events contain exactly one skipped test named
    `skippedTest` with `"for demonstration purposes"` as the _reason_.

If you want to verify the type of exception thrown from the `failingTest()` method in
<<testkit-engine-ExampleTestCase, `ExampleTestCase`>>, you can do that as follows.

[TIP]
====
For details on what _conditions_ are available for use with AssertJ assertions against
events and execution results, consult the Javadoc for `{EventConditions}` and
`{TestExecutionResultConditions}`, respectively.
====

[source,java,indent=0]
----
include::{testDir}/example/testkit/EngineTestKitFailedMethodDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<testkit-engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _test_ events.
<5> Assert that the recorded _test_ events contain exactly one failing test named
    `failingTest` with an exception of type `ArithmeticException` and an error message
    equal to `"/ by zero"`.

Although typically unnecessary, there are times when you need to verify **all** of the
events fired during the execution of a `TestPlan`. The following test demonstrates how to
achieve this via the `assertEventsMatchExactly()` method in the `EngineTestKit` API.

[TIP]
====
Since `assertEventsMatchExactly()` matches conditions exactly in the order in which the
events were fired, <<testkit-engine-ExampleTestCase, `ExampleTestCase`>> has been
annotated with `@TestMethodOrder(OrderAnnotation.class)` and each test method has been
annotated with `@Order(...)`. This allows us to enforce the order in which the test
methods are executed, which in turn allows our `verifyAllJupiterEvents()` test to be
reliable.
====

If you want to do a _partial_ match _with_ or _without_ ordering requirements, you can use
the methods `assertEventsMatchLooselyInOrder()` and `assertEventsMatchLoosely()`,
respectively.

[source,java,indent=0]
----
include::{testDir}/example/testkit/EngineTestKitAllEventsDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<testkit-engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _all_ events.
<5> Print all events to the supplied `writer` for debugging purposes. Debug information
    can also be written to an `OutputStream` such as `System.out` or `System.err`.
<6> Assert _all_ events in exactly the order in which they were fired by the test engine.

The `debug()` invocation from the preceding example results in output similar to the
following.

[source,options="nowrap"]
----
All Events:
	Event [type = STARTED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.082280Z, payload = null]
	Event [type = STARTED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.089339Z, payload = null]
	Event [type = SKIPPED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:skippedTest()], timestamp = 2018-12-14T12:45:14.094314Z, payload = 'for demonstration purposes']
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.095182Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.104922Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.106121Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.109956Z, payload = TestExecutionResult [status = ABORTED, throwable = org.opentest4j.TestAbortedException: Assumption failed: abc does not contain Z]]
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.110680Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.111217Z, payload = TestExecutionResult [status = FAILED, throwable = java.lang.ArithmeticException: / by zero]]
	Event [type = FINISHED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.113731Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
	Event [type = FINISHED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.113806Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
----