Repository: pytorch
Issue: ROCm distributed flaky on test_distributed_spawn
Labels: module: rocm,triaged

### 🐛 Describe the bug

`test_distributed_spawn` flakily failes on ROCm distributed when running with the nccl backend. 
Examples:
https://hud.pytorch.org/pytorch/pytorch/commit/75cb99e54971c0fdb5f1a6516e3c366af998be43#11706358342
https://hud.pytorch.org/pytorch/pytorch/commit/fafb410985d2cb94bd95f12f0c392bad9385b643#11668037807
https://hud.pytorch.org/pytorch/pytorch/commit/05943712a443138497c185405b575043b2916f34#11649638366

<details><summary>Long example from copied from logs</summary>

```

2023-02-28T13:46:36.9960944Z Running tests...
2023-02-28T13:46:36.9961599Z ----------------------------------------------------------------------
2023-02-28T13:46:36.9962850Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3738
2023-02-28T13:46:36.9963852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3739
2023-02-28T13:46:36.9964720Z [W Module.cpp:1170] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator())
2023-02-28T13:46:36.9965866Z /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:119: UserWarning: loaded 64 slow tests
2023-02-28T13:46:36.9966608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-02-28T13:46:36.9967591Z /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 256 disabled tests
2023-02-28T13:46:36.9968422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-02-28T13:46:36.9969163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-02-28T13:46:36.9970321Z /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:119: UserWarning: loaded 64 slow tests
2023-02-28T13:46:36.9971041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-02-28T13:46:36.9971970Z /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 256 disabled tests
2023-02-28T13:46:36.9972789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-02-28T13:46:36.9973468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-02-28T13:46:36.9974508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-02-28T13:46:36.9975618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-02-28T13:46:36.9976481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-02-28T13:46:36.9977244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-02-28T13:46:36.9978052Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-02-28T13:46:36.9979376Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-02-28T13:46:36.9980534Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-02-28T13:46:36.9981908Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-02-28T13:46:36.9983159Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
2023-02-28T13:46:36.9984024Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
2023-02-28T13:46:36.9984640Z free(): double free detected in tcache 2
2023-02-28T13:46:36.9985403Z INFO:torch.testing._internal.common_distributed:Received event Event.GET_TRACEBACK on process 0
2023-02-28T13:46:36.9986360Z INFO:torch.testing._internal.common_distributed:Received event Event.GET_TRACEBACK on process 1
2023-02-28T13:46:36.9987161Z INFO:torch.testing._internal.common_distributed:Process 0 sent traceback
2023-02-28T13:46:36.9987906Z INFO:torch.testing._internal.common_distributed:Process 1 sent traceback
2023-02-28T13:46:36.9988637Z ERROR:torch.testing._internal.common_distributed:Process 0 timed out with traceback: 
2023-02-28T13:46:36.9989049Z 
2023-02-28T13:46:36.9989289Z Current thread 0x00007f72f2ffe700 (most recent call first):
2023-02-28T13:46:36.9990272Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 620 in _event_listener
2023-02-28T13:46:36.9991158Z   File "/opt/conda/envs/py_3.8/lib/python3.8/threading.py", line 870 in run
2023-02-28T13:46:36.9992098Z   File "/opt/conda/envs/py_3.8/lib/python3.8/threading.py", line 932 in _bootstrap_inner
2023-02-28T13:46:36.9992827Z   File "/opt/conda/envs/py_3.8/lib/python3.8/threading.py", line 890 in _bootstrap
2023-02-28T13:46:36.9993287Z 
2023-02-28T13:46:36.9993565Z Thread 0x00007f7aa711e180 (most recent call first):
2023-02-28T13:46:36.9994629Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1731 in all_reduce
2023-02-28T13:46:36.9995711Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1480 in wrapper
2023-02-28T13:46:36.9996960Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/algorithms/model_averaging/utils.py", line 36 in average_parameters
2023-02-28T13:46:36.9998306Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/algorithms/model_averaging/utils.py", line 71 in average_parameters_or_parameter_groups
2023-02-28T13:46:36.9999510Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/algorithms/model_averaging/averagers.py", line 118 in average_parameters
2023-02-28T13:46:37.0000948Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 1129 in test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager
2023-02-28T13:46:37.0002168Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 174 in wrapper
2023-02-28T13:46:37.0003183Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 543 in wrapper
2023-02-28T13:46:37.0004267Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 657 in run_test
2023-02-28T13:46:37.0005390Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 581 in _run
2023-02-28T13:46:37.0006269Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/process.py", line 108 in run
2023-02-28T13:46:37.0007000Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap
2023-02-28T13:46:37.0007775Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/spawn.py", line 129 in _main
2023-02-28T13:46:37.0008461Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/spawn.py", line 116 in spawn_main
2023-02-28T13:46:37.0009058Z   File "<string>", line 1 in <module>
2023-02-28T13:46:37.0009379Z 
2023-02-28T13:46:37.0009694Z ERROR:torch.testing._internal.common_distributed:Process 1 timed out with traceback: 
2023-02-28T13:46:37.0010067Z 
2023-02-28T13:46:37.0010329Z Current thread 0x00007efa0577e700 (most recent call first):
2023-02-28T13:46:37.0011220Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 620 in _event_listener
2023-02-28T13:46:37.0012063Z   File "/opt/conda/envs/py_3.8/lib/python3.8/threading.py", line 870 in run
2023-02-28T13:46:37.0012816Z   File "/opt/conda/envs/py_3.8/lib/python3.8/threading.py", line 932 in _bootstrap_inner
2023-02-28T13:46:37.0013668Z   File "/opt/conda/envs/py_3.8/lib/python3.8/threading.py", line 890 in _bootstrap
2023-02-28T13:46:37.0014071Z 
2023-02-28T13:46:37.0014293Z Thread 0x00007efeb4eab180 (most recent call first):
2023-02-28T13:46:37.0015247Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1731 in all_reduce
2023-02-28T13:46:37.0016340Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1480 in wrapper
2023-02-28T13:46:37.0017471Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/algorithms/model_averaging/utils.py", line 36 in average_parameters
2023-02-28T13:46:37.0018885Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/algorithms/model_averaging/utils.py", line 71 in average_parameters_or_parameter_groups
2023-02-28T13:46:37.0020128Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/distributed/algorithms/model_averaging/averagers.py", line 118 in average_parameters
2023-02-28T13:46:37.0021610Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 1129 in test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager
2023-02-28T13:46:37.0022816Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 174 in wrapper
2023-02-28T13:46:37.0023959Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 543 in wrapper
2023-02-28T13:46:37.0025026Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 657 in run_test
2023-02-28T13:46:37.0026126Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 581 in _run
2023-02-28T13:46:37.0026956Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/process.py", line 108 in run
2023-02-28T13:46:37.0027733Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap
2023-02-28T13:46:37.0028534Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/spawn.py", line 129 in _main
2023-02-28T13:46:37.0029353Z   File "/opt/conda/envs/py_3.8/lib/python3.8/multiprocessing/spawn.py", line 116 in spawn_main
2023-02-28T13:46:37.0029981Z   File "<string>", line 1 in <module>
2023-02-28T13:46:37.0030286Z 
2023-02-28T13:46:37.0030473Z ERROR (300.600s)
2023-02-28T13:46:37.0031231Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... Timing out after 300 seconds and killing subprocesses.
2023-02-28T13:46:37.0032297Z     test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager errored - num_retries_left: 3
2023-02-28T13:46:37.0032769Z Traceback (most recent call last):
2023-02-28T13:46:37.0033628Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 541, in wrapper
2023-02-28T13:46:37.0034254Z     self._join_processes(fn)
2023-02-28T13:46:37.0035187Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 759, in _join_processes
2023-02-28T13:46:37.0035910Z     self._check_return_codes(elapsed_time)
2023-02-28T13:46:37.0036873Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_distributed.py", line 809, in _check_return_codes
2023-02-28T13:46:37.0037545Z     raise RuntimeError(
2023-02-28T13:46:37.0038061Z RuntimeError: Process 0 terminated or timed out after 300.0667040348053 seconds

```
</details>


I believe this is runner related as I only see it happen on 
* worker-rocm-amd-50
* worker-rocm-amd-60
* worker-rocm-amd-64
* worker-rocm-amd-66
* worker-rocm-amd-70
* worker-rocm-amd-106
* worker-rocm-amd-118

50, 60, 64, 66, and 70 are especially bad, they have each run the the distributed test config at least 15 times on master and have succeeded 0 times.  This makes me think that these workers also fail in other ways on other distributed tests, but I haven't investigated that much.

The first known bad is https://hud.pytorch.org/pytorch/pytorch/commit/2b0d7e63f0c146152ec4786fe8799ce2ec17fe7c#11073989965, which is a few days after a month of ROCm distributed being disabled.


### Versions

CI

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport