Repository: pytorch
Issue: Segmentation fault in forward pass using DataParallel and multiple GPUs
Labels: module: cuda,triaged,module: data parallel

## 🐛 Bug

When I want to use DataParallel to run my model on multiple GPUs I encounter a segmentation fault. A small snippet reproducing the error is below. Running the model on only 1 GPU works.

## To Reproduce

The following fails for me:
```
# SegmentationFault.py

import torch
import torch.nn as nn
import faulthandler
faulthandler.enable()

device = "cuda:0"

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.lr = nn.Linear(10,1)
    
    def forward(self, x):
        print(x.shape)
        return self.lr(x)
    
inp = torch.randn(100,10).to(device)

model = Model().to(device)
model = nn.DataParallel(model)
out = model(inp)
print(out.shape)
(out.sum()-5).backward()
```

Output:
```
Fatal Python error: Segmentation fault

Current thread 0x00007f7a19414740 (most recent call first):
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/cuda/comm.py", line 39 in broadcast_coalesced
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/nn/parallel/_functions.py", line 21 in forward
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/nn/parallel/replicate.py", line 72 in _broadcast_coalesced_reshape
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/nn/parallel/replicate.py", line 89 in replicate
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 159 in replicate
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 154 in forward
  File "/home/x/anaconda3/envs/GPU-Test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550 in __call__
  File "SegmentationFault.py", line 29 in <module>
Segmentation fault

```


<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

Should run without segmentation fault.

## Environment
```
PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Debian GNU/Linux 10 (buster)
GCC version: (Debian 8.3.0-6) 8.3.0
CMake version: version 3.13.4

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 9.2.148
GPU models and configuration:
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000

Nvidia driver version: 440.82
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.5.1
[pip3] torchvision==0.6.0a0+35d732a
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.1                      217
[conda] mkl-service               2.3.0            py38he904b0f_0
[conda] mkl_fft                   1.1.0            py38h23d657b_0
[conda] mkl_random                1.1.1            py38h0573a6f_0
[conda] pytorch                   1.5.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.6.1                py38_cu101    pytorch
```

## Additional context
nvidia-smi
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82       Driver Version: 440.82       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro RTX 8000     On   | 00000000:87:00.0 Off |                  Off |
| 33%   20C    P8     4W / 260W |      0MiB / 48601MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro RTX 8000     On   | 00000000:C4:00.0 Off |                  Off |
| 33%   24C    P8     9W / 260W |      0MiB / 48601MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

cc @ngimel