Repository: pytorch
Issue: DISABLED test_fully_shard_post_optim_event_overlap (__main__.TestFullyShardOverlap)
Labels: oncall: distributed,module: flaky-tests,skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_fully_shard_post_optim_event_overlap&suite=TestFullyShardOverlap&limit=100) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/27633006588).

Over the past 3 hours, it has been determined flaky in 13 workflow(s) with 25 failures and 13 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_fully_shard_post_optim_event_overlap`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.



<details><summary>Sample error message</summary>

```
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 538, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 761, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 810, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 540, in wrapper
    fn()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2829, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 181, in wrapper
    return func(*args, **kwargs)
  File "/var/lib/jenkins/workspace/test/distributed/_composable/fsdp/test_fully_shard_overlap.py", line 142, in test_fully_shard_post_optim_event_overlap
    self.assertGreater(baseline_time, expected_test_time)
  File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 1244, in assertGreater
    self.fail(self._formatMessage(msg, standardMsg))
  File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 675, in fail
    raise self.failureException(msg)
AssertionError: 391.5706787109375 not greater than 405

To execute this test, run the following from the base repo dir:
    python test/distributed/_composable/fsdp/test_fully_shard_overlap.py -k TestFullyShardOverlap.test_fully_shard_post_optim_event_overlap

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 540, in wrapper
    fn()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2829, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 181, in wrapper
    return func(*args, **kwargs)
  File "/var/lib/jenkins/workspace/test/distributed/_composable/fsdp/test_fully_shard_overlap.py", line 142, in test_fully_shard_post_optim_event_overlap
    self.assertGreater(baseline_time, expected_test_time)
  File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 1244, in assertGreater
    self.fail(self._formatMessage(msg, standardMsg))
  File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 675, in fail
    raise self.failureException(msg)
AssertionError: 391.4037170410156 not greater than 405

To execute this test, run the following from the base repo dir:
    python test/distributed/_composable/fsdp/test_fully_shard_overlap.py -k TestFullyShardOverlap.test_fully_shard_post_optim_event_overlap

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```

</details>


Test file path: `distributed/_composable/fsdp/test_fully_shard_overlap.py`

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @clee2000