Repository: pytorch
Issue: Pytorch 1.7.1 hangs with multi-gpu, while Pytorch 1.6.0 works correctly
Labels: triaged,module: deadlock,module: data parallel

## 🐛 Bug

Basic `torch.nn.DataParallel` usage succeeds on Pytorch `1.6.0` using both CUDA `10.2.89` and `11.0.3`. However using the appropriate Pytorch `1.7.1` version with each of these CUDA versions, the python process hangs indefinitely and has to be killed. The complete version combinations tested and outcomes are listed below.

The Nvidia driver version is `450.51.06` throughout and two Tesla V100 GPU's are used.

![cuda-torch-versions](https://user-images.githubusercontent.com/13351115/103912165-9f114300-510f-11eb-82ed-21730fb10951.png)

## To Reproduce

Steps to reproduce the behavior:

1. Setup the hardware and software environment as above
2. Run the below script using `pytest` (adapted from https://github.com/pytorch/tutorials/blob/master/beginner_source/blitz/data_parallel_tutorial.py)

```python
"""
Requires: pytest, pytest-timeout
"""
import pytest
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader


class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class Model(nn.Module):
    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc = nn.Linear(input_size, output_size)

    def forward(self, input):
        output = self.fc(input)
        print(f"In Model: input size {input.size()}, output size {output.size()}")

        return output


def test_cuda__is_available():
    assert torch.cuda.is_available(), "Cuda is not available"


def test_cuda__device_count():
    device_count = torch.cuda.device_count()
    assert device_count == 2, f"Detected {device_count} devices; expected 2"


@pytest.mark.timeout(20)
def test_dataparallel__two_gpus(capsys):
    input_size = 5
    output_size = 2
    batch_size = 20
    data_size = 100
    per_gpu_batch_size = batch_size // 2

    device = torch.device("cuda:0")
    rand_loader = DataLoader(
        dataset=RandomDataset(input_size, data_size),
        batch_size=batch_size,
        shuffle=True,
    )

    model = Model(input_size, output_size)
    model = nn.DataParallel(model)
    model.to(device)

    per_gpu_expected_output = (
        f"In Model: input size torch.Size([{per_gpu_batch_size}, {input_size}])"
    )
    per_gpu_expected_output += (
        f", output size torch.Size([{per_gpu_batch_size}, {output_size}])"
    )
    expected_output = f"{per_gpu_expected_output}\n{per_gpu_expected_output}\n"
    for data in rand_loader:
        model_input = data.to(device)
        model_output = model(model_input)
        assert (model_input.size()[0] == batch_size) and (
            model_output.size()[0] == batch_size
        )

        captured = capsys.readouterr()
        assert captured.out == expected_output
```

## Expected behavior

We would expect all tests to pass when the cuda and Pytorch versions are aligned correctly. However, all the Pytorch `1.6.0` tests pass while the Pytorch `1.7.1` `DataParallel` tests (`test_dataparallel__two_gpus`) end up hanging and failing.

## Environment

![collect-env-output](https://user-images.githubusercontent.com/13351115/103914165-16e06d00-5112-11eb-958d-71f41e5b6448.png)

 - PyTorch Version: 1.7.1
 - OS: Ubuntu 18.04.3
 - How you installed PyTorch: pip
 - Build command you used (if compiling from source): NA
 - Python version: 3.7.9
 - CUDA/cuDNN version: 11.0
 - GPU models and configuration: Tesla V100
 - Any other relevant information: Running on Amazon EKS
    - EKS version: 1.6
    - EKS GPU node AMI: ami-038f513cc4c905c3f
    - EKS GPU host CUDA version: 11.0.221