Repository: pytorch
Issue: test_profiler_with_remote_builtin (__main__.TensorPipeRpcTestWithSpawn) is flaky
Labels: oncall: distributed,triaged,module: flaky-tests,module: rpc

https://app.circleci.com/pipelines/github/pytorch/pytorch/263723/workflows/86717ae9-14ff-4bfd-bcd5-973b53b01c1e/jobs/10367347/steps

Seems to be rare, not disabling yet. 

```
Jan 25 22:26:38 ======================================================================
Jan 25 22:26:38 ERROR [1.540s]: test_profiler_with_remote_builtin (__main__.TensorPipeRpcTestWithSpawn)
Jan 25 22:26:38 ----------------------------------------------------------------------
Jan 25 22:26:38 Traceback (most recent call last):
Jan 25 22:26:38   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 282, in wrapper
Jan 25 22:26:38     self._join_processes(fn)
Jan 25 22:26:38   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 399, in _join_processes
Jan 25 22:26:38     self._check_return_codes(elapsed_time)
Jan 25 22:26:38   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 435, in _check_return_codes
Jan 25 22:26:38     raise RuntimeError(error)
Jan 25 22:26:38 RuntimeError: Processes 1 exited with error code 10
Jan 25 22:26:38 
Jan 25 22:26:38 ----------------------------------------------------------------------
```

```
Jan 25 22:18:37   test_profiler_with_remote_builtin (__main__.TensorPipeRpcTestWithSpawn) ... ERROR:root:Caught exception: 
Jan 25 22:18:37 Traceback (most recent call last):
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 285, in wrapper
Jan 25 22:18:37     fn()
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/dist_utils.py", line 90, in new_test_method
Jan 25 22:18:37     return_value = old_test_method(self, *arg, **kwargs)
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 1712, in test_profiler_with_remote_builtin
Jan 25 22:18:37     self._run_test_profiler_with_remote_builtin()
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 1700, in _run_test_profiler_with_remote_builtin
Jan 25 22:18:37     use_record_function=True
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 1609, in _profiler_test_with_rpc
Jan 25 22:18:37     self.assertLess(foo_event_ix, rpc_event_idx)
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/unittest/case.py", line 1209, in assertLess
Jan 25 22:18:37     self.fail(self._formatMessage(msg, standardMsg))
Jan 25 22:18:37   File "/opt/conda/lib/python3.6/unittest/case.py", line 670, in fail
Jan 25 22:18:37     raise self.failureException(msg)
Jan 25 22:18:37 AssertionError: 1 not less than 0
Jan 25 22:18:37 exiting process with exit code: 10
Jan 25 22:18:38 [W tensorpipe_agent.cpp:648] RPC agent for worker2 encountered error when reading incoming request from worker1: EOF: end of file (this is expected to happen during shutdown)
Jan 25 22:18:38 [W tensorpipe_agent.cpp:648] RPC agent for worker0 encountered error when reading incoming request from worker1: EOF: end of file (this is expected to happen during shutdown)
Jan 25 22:18:38 Process 1 terminated with exit code 10, terminating remaining processes.
Jan 25 22:18:38 ERROR (1.540s)
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @rohan-varma @jjlilley @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu