Repository: pytorch
Issue: Summary of tests in machine readable format
Labels: module: tests,triaged

### 🐛 Describe the bug

We are running the test suite `python run_test.py --continue-through-error` after installing PyTorch on our (many) clusters. For that we need to know how many tests were run, succeeded and failed to make an assessment whether the installation is working at all and especially in that environment. Many of the issues I reported and PRs I submitted are based on that.

Until recently we were able to parse the build output containing summary lines such as
```
    # ===================== 2 failed, 128 passed, 2 skipped, 2 warnings in 3.43s =====================
    # test_quantization failed!
```

This is easy to grep for and even extract manually. However recently (2.3.0 release) we get something like (I'll file a separate bug report for that failure once I know more, but the specific failure is only an example here):

```
=======================short test summary info =======================
FAILED [0.0003s] export/test_lift_unlift.py::TestLift::test_duplicate_constant_access - OSError: /caffe2/test/cpp/jit:test_custom_class_registrations: cannot open shared object file: No such file or directory
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================= 1 failed, 2 rerun in 0.07s =======================
Got exit code 1
Retrying...
[above repeated a few times]
Got exit code 1
Retrying...
======================short test summary info ======================
FAILED [0.0003s] export/test_lift_unlift.py::TestLift::test_unlift_nonpersistent_buffer - OSError: /caffe2/test/cpp/jit:test_custom_class_registrations: cannot open shared object file: No such file or dire...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================== 1 failed, 3 deselected, 2 rerun in 0.07s =============
Got exit code 1
Retrying...
[Similar until]
==================== 5 deselected in 0.06s ====================
The following tests failed consistently: ['test/export/test_lift_unlift.py::TestLift::test_duplicate_constant_access', 'test/export/test_lift_unlift.py::TestLift::test_lift_basic', 'test/export/test_lift_unlift.py::TestLift::test_lift_nested', 'test/export/test_lift_unlift.py::TestLift::test_unlift_nonpersistent_buffer', 'test/export/test_lift_unlift.py::ConstantAttrMapTest::test_dict_api']
export/test_lift_unlift 1/1 failed!

```

This is mostly due to 3b7d60b6ffef34c1acc35a6f43b1fde22bf8f27e which removed the following and replaced the whole logic by a custom pytest plugin rerunning differently (basically: Manually rerun the suite continuing after the first failure until the next, then repeat):
```
    elif options.continue_through_error:
        # If continue through error, don't stop on first failure
        rerun_options = ["--reruns=2"]
[Other test]
==================49 passed, 4 skipped, 41 deselected in 7.04s ==================
```

So that cannot be reasonably parsed check for the number of failures or the number of tests anymore.
Additionally, as tests are run in parallel, partial outputs from different tests appear after each other making it virtually impossible to attribute some output to a specific test.

I have seen the option `--save-xml` which seems to be intended for such purposes but that isn't passed by `run_test.py` to the tests spawned by it. Hence reporting this here as a bug instead of only a feature request.

The main question: Is there any reliable way to extract the number of failed and run tests after running `run_test.py`?

### Versions

PyTorch 2.2.0-2.3.1 and main as of now

cc @mruberry @ZainRizvi