Repository: pytorch
Issue: ProcessGroupGlooTest.test_gather_stress is flaky
Labels: oncall: distributed,triaged,module: flaky-tests,module: c10d

Running [this pytorch-linux-trusty-py3.6-gcc5.4 CI docker image](308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-trusty-py3.6-gcc5.4:282-8f19dfe947a78822ca7bf490d2f375cf2800c0d0) locally, I occasionally get
```
test_gather_stress (__main__.ProcessGroupGlooTest) ... Process process 3:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test_c10d.py", line 471, in _run
    getattr(self, self.id().split(".")[2])()
  File "test_c10d.py", line 437, in wrapper
    fn(self)
  File "test_c10d.py", line 985, in test_gather_stress
    self._test_gather_stress(inputs, lambda t: t.clone())
  File "test_c10d.py", line 973, in _test_gather_stress
    work_handle.wait()
RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:119] Timed out waiting 1000ms for send operation to complete
Process process 2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test_c10d.py", line 471, in _run
    getattr(self, self.id().split(".")[2])()
  File "test_c10d.py", line 437, in wrapper
    fn(self)
  File "test_c10d.py", line 985, in test_gather_stress
    self._test_gather_stress(inputs, lambda t: t.clone())
  File "test_c10d.py", line 973, in _test_gather_stress
    work_handle.wait()
RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:119] Timed out waiting 1000ms for send operation to complete
Process process 0:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test_c10d.py", line 471, in _run
    getattr(self, self.id().split(".")[2])()
  File "test_c10d.py", line 437, in wrapper
    fn(self)
  File "test_c10d.py", line 985, in test_gather_stress
    self._test_gather_stress(inputs, lambda t: t.clone())
  File "test_c10d.py", line 973, in _test_gather_stress
    work_handle.wait()
RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:119] Timed out waiting 1000ms for send operation to complete
Process process 1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test_c10d.py", line 471, in _run
    getattr(self, self.id().split(".")[2])()
  File "test_c10d.py", line 437, in wrapper
    fn(self)
  File "test_c10d.py", line 985, in test_gather_stress
    self._test_gather_stress(inputs, lambda t: t.clone())
  File "test_c10d.py", line 973, in _test_gather_stress
    work_handle.wait()
RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/pair.cc:534] Read error [127.0.0.1]:37100: Connection reset by peer
FAIL
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang