Repository: pytorch
Issue: Intermittent segmentation fault when using rpc_async/rpc_sync with CUDA tensors
Labels: oncall: distributed,triaged

## 🐛 Bug

`rpc_sync` (and `rpc_async`) with CUDA tensors result in a segmentation fault at regular intervals. I am not able to reproduce this consistently but happens often enough. 
The error is raised when using `rpc_sync` (and `rpc_async`) for P2P communication between 2 processes on the same machine as well as on remote machines.

Stack Trace:

```
[W tensorpipe_agent.cpp:983] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:260)
[W tensorpipe_agent.cpp:779] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:260)
run_worker 0 with world size 2
Traceback (most recent call last):
  File "playground/herring/rpc_perf.py", line 214, in <module>
    mp.spawn(run_worker, args=(args, 2), nprocs=2, join=True)
  File "/private/home/anj/.conda/envs/test_clone/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/private/home/anj/.conda/envs/test_clone/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/private/home/anj/.conda/envs/test_clone/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 130, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGSEGV

```

## To Reproduce
Run the following snippet of code with the --use_cuda_tensors flag.
https://gist.github.com/anj-s/0af840dfe2e9e192c66760a9da59e951

## Expected behavior
There should not be a segmentation fault.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).


You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
Collecting environment information...
PyTorch version: 1.9.0.dev20210428+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration:
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.9.0.dev20210428+cu111
[pip3] torchaudio==0.9.0.dev20210428
[pip3] torchtext==0.10.0a0+34ccda8
[pip3] torchvision==0.10.0.dev20210428+cu111
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.0.221             h6bb024c_0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.2.0            py38h23d657b_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] numpy                     1.19.2           py38h54aff64_0
[conda] numpy-base                1.19.2           py38hfa32c7d_0
[conda] torch                     1.9.0.dev20210428+cu111           <pip>
[conda] torchaudio                0.9.0.dev20210428           <pip>
[conda] torchtext                 0.10.0a0+34ccda8           <pip>
[conda] torchvision               0.10.0.dev20210428+cu111           <pip>
## Additional context

<!-- Add any other context about the problem here. -->


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23