Repository: pytorch
Issue: [ROCm] test_gather_stress_cuda is flaky
Labels: module: rocm,triaged,module: flaky-tests

```
6:06:39 test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... Caught exception: 
06:06:42 Traceback (most recent call last):
06:06:42   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 516, in run_test
06:06:42     getattr(self, test_name)()
06:06:42   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 400, in wrapper
06:06:42     fn()
06:06:42   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 93, in wrapper
06:06:42     return func(*args, **kwargs)
06:06:42   File "distributed/test_c10d_gloo.py", line 1049, in test_gather_stress_cuda
06:06:42     self._test_gather_stress(inputs, lambda t: t.clone().cuda())
06:06:42   File "distributed/test_c10d_gloo.py", line 1039, in _test_gather_stress
06:06:42     msg=("Mismatch in iteration %d for root %d" % (iter, root)),
06:06:42   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1395, in assertEqual
06:06:42     exact_dtype=exact_dtype, exact_device=exact_device)
06:06:42   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1395, in assertEqual
06:06:42     exact_dtype=exact_dtype, exact_device=exact_device)
06:06:42   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1361, in assertEqual
06:06:42     super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
06:06:42   File "/opt/conda/lib/python3.6/unittest/case.py", line 682, in assertTrue
06:06:42     raise self.failureException(msg)
06:06:42 AssertionError: False is not true : Tensors failed to compare as equal!Found 1 different element(s) (out of 1), with the greatest difference of 186 (280 vs. 94) occuring at index 0.
06:06:42 Mismatch in iteration 278 for root 3
06:06:42  exiting process with exit code: {MultiProcessTestCase.TEST_ERROR_EXIT_CODE}
06:06:42 ERROR

06:07:01 ======================================================================
06:07:01 ERROR: test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
06:07:01 ----------------------------------------------------------------------
06:07:01 Traceback (most recent call last):
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 398, in wrapper
06:07:01     self._join_processes(fn)
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 590, in _join_processes
06:07:01     self._check_return_codes(elapsed_time)
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 633, in _check_return_codes
06:07:01     raise RuntimeError(error)
06:07:01 RuntimeError: Process 3 exited with error code 10 and exception:
06:07:01 Traceback (most recent call last):
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 516, in run_test
06:07:01     getattr(self, test_name)()
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 400, in wrapper
06:07:01     fn()
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 93, in wrapper
06:07:01     return func(*args, **kwargs)
06:07:01   File "distributed/test_c10d_gloo.py", line 1049, in test_gather_stress_cuda
06:07:01     self._test_gather_stress(inputs, lambda t: t.clone().cuda())
06:07:01   File "distributed/test_c10d_gloo.py", line 1039, in _test_gather_stress
06:07:01     msg=("Mismatch in iteration %d for root %d" % (iter, root)),
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1395, in assertEqual
06:07:01     exact_dtype=exact_dtype, exact_device=exact_device)
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1395, in assertEqual
06:07:01     exact_dtype=exact_dtype, exact_device=exact_device)
06:07:01   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1361, in assertEqual
06:07:01     super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
06:07:01   File "/opt/conda/lib/python3.6/unittest/case.py", line 682, in assertTrue
06:07:01     raise self.failureException(msg)
06:07:01 AssertionError: False is not true : Tensors failed to compare as equal!Found 1 different element(s) (out of 1), with the greatest difference of 186 (280 vs. 94) occuring at index 0.
06:07:01 Mismatch in iteration 278 for root 3
```

https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-linux-bionic-rocm4.2-py3.6-test1/1860/console

cc @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23