Repository: assertj/assertj

Issue #3550:  IntelliJ IDEA run window fails to show all SoftAssertions errors in concurrent JUnit tests
Created: 2024-07-25T14:57:27Z
Describe the bug
In my current test setup, I am using SoftAssertions within nested classes. After running such a test class with JUnit concurrency, individual test failures in the IntelliJ Run window do not correctly display. Although all assertion errors are logged, only one test failure is shown.

assertj core version: 3.26.3
java version: 22
test framework version: (JUnit 5) 5.11.0-M2
IntelliJ IDEA 2024.1.4 (Ultimate Edition)
os (if relevant): Windows 11

Test case reproducing the bug

Set the concurrency run parameters for JUnit.
Run the test class SoftAssertInConcurrentRunOfTests.
Observe the logs to see that three assertion errors are captured.
Check the IntelliJ Run window to see that only one test is marked as failed.

Concurrency run parameters:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.config.strategy= dynamic
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
import org.assertj.core.api.SoftAssertions;
import org.assertj.core.api.junit.jupiter.InjectSoftAssertions;
import org.assertj.core.api.junit.jupiter.SoftAssertionsExtension;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith({SoftAssertionsExtension.class})
public class SoftAssertInConcurrentRunOfTests {

    @InjectSoftAssertions
    private SoftAssertions softly;

    @Test
    void testInsideSimpleClass() {
        softly.assertThat(1).isEqualTo(2);
    }

    @Nested
    class NestedTestClass {
        @Test
        void testInsideNestedClass1() {
            softly.assertThat(1).isEqualTo(2);
        }
    }

    @Nested
    class NestedTestClass2 {
        @Test
        void testInsideNestedClass2() {
            softly.assertThat(1).isEqualTo(2);
        }
    }
}

The problem does not occur when SoftAssertions are injected individually for each test.
@Nested
    class NestedTestClass2 {
        @Test
        void testInsideNestedClass2(SoftAssertions softly) {
            softly.assertThat(1).isEqualTo(2);
        }
    }
If it is intended way how it works, it might be helpful to update documentation for usage of SoftAssertions with junit concurency/nested test classes.

Duplicate issue #2356: `@InjectSoftAssertions` doesn't work as expected when parallel mode is enabled
Created: 2021-10-01T13:01:38Z
Summary
Framework: Junit 5
Using InjectSoftAssertions and running tests in concurrent mode are leading to incorrect test results. Tests that should have failed are eventually passed.
Example
@ExtendWith(SoftAssertionsExtension.class)
public class SoftAssertionTest {

    @InjectSoftAssertions private SoftAssertions softAssertions;

    @Test
    @DisplayName("Test 1")
    void test1() {
        softAssertions.assertThat("Test 1").as("Test 1").isEqualTo("Fail");
    }

    @Test
    @DisplayName("Test 2")
    void test2() {
        softAssertions.assertThat("Test 2").as("Test 2").isEqualTo("Fail");
    }
}
Both tests should fail, but one of them passes and the other fails with two errors(from test one and test two).
Screenshot from IDE

If i disable parallel mode in the junit-platfrom.properties or set these two annotations
    @TestInstance(TestInstance.Lifecycle.PER_CLASS)
    @Execution(ExecutionMode.CONCURRENT)
it will help. But this is not a solution for me.
Repository to reproduce the issue. You can run tests by maven or from IDE. The result will be the same.
https://github.com/Rusgallaz/SoftAssertionConcurrent
Version
assertj-core  3.21.0

Closing comment (2024-10-27T12:35:29Z): This is similar to #3550. It appears that a single test thread entering afterTestExecution collects all assertion errors and reports them together, while other tests with no errors are shown as passing (green).
If handled correctly, this could resolve #3550, though I’m not sure of the exact approach.
One simple solution would be to use the thread-local collector (currently used for isPerClassConcurrent(ctx)) more broadly — to cover all concurrent cases (also per method + concurrent). However, I'm not sure of all the possible implications of this change.