Repository: pytorch
Issue: Using Tensor.to(device) after distributed all_reduce intermittently causes deadlock with NCCL 
Labels: oncall: distributed,triaged,module: nccl

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce
Hello, I'm a beginner of pyTorch, and learning it with [pytorch tutorials](https://pytorch.org/tutorials/)

and at [Writing Distributed Applications with PyTorch](https://pytorch.org/tutorials/intermediate/dist_tuto.html), I am trying to train Resnet-50 with CIFAR-10 Datasets and 4 GPUs (Tesla P40).

However, when I used NCCL backend, it causes deadlock intermittently. ( it worked well about 1 out of 5 times )

I'm not sure if it is a bug or not.

```python
import os

import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
import torchvision.datasets as datasets
import torchvision.models as models
import torchvision.transforms as transforms


def run(rank, size):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29502'
    device = torch.device(f'cuda:{rank}')

    dist.init_process_group('nccl', rank=rank, world_size=size)

    model = models.resnet50().to(device)
    optimizer = optim.SGD(model.parameters(), lr=0.001)
    loss_fn = nn.CrossEntropyLoss()
    dataloader = torch.utils.data.DataLoader(
        datasets.CIFAR10('/data/private/datasets', transform=transforms.ToTensor(), train=True, download=True),
        batch_size=1024)
    # model = nn.parallel.DistributedDataParallel(model, device_ids=[device])
    for inputs, labels in dataloader:
        optimizer.zero_grad()
        inputs = inputs.to(device) # all processes are locked here at the second iteration
        labels = labels.to(device)
        loss_fn(model(inputs), labels).backward()
        for param in model.parameters():
            dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
            param.grad.data /= size
        optimizer.step()
    print('done')


if __name__ == '__main__':
    world_size = 4
    mp.spawn(run, args=(world_size,), nprocs=world_size, join=True)

```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

```
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
done
done
done
done
```
<!-- A clear and concise description of what you expected to happen. -->

## Environment
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40

Nvidia driver version: 418.43
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.17.2
[pip3] torch==1.3.1
[pip3] torchvision==0.4.2
[conda] _pytorch_select           0.2                       gpu_0  
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py36he904b0f_0  
[conda] mkl_fft                   1.0.15           py36ha843d7b_0  
[conda] mkl_random                1.1.0            py36hd6b4f25_0  
[conda] pytorch                   1.3.1           cuda100py36h53c1284_0  
[conda] torchvision               0.4.2           cuda100py36hecfc37a_0

NCCL version 2.4.8+cuda10.0
## Additional context

<!-- Add any other context about the problem here. -->

I tried training several times in different enviroment, and it worked well in the following conditions

1. running with Gloo backend.
2. using `nn.parallel.DistributedDataParallel(model)`
3. without using `average_gradients`



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar