Repository: pytorch
Issue: pytorch DDP hangs at .backward() call
Labels: needs reproduction,triaged,module: deadlock,module: ddp

## 🐛 Bug

Training pytorch DDP on 8 GPUs with NCCL backend hangs at .backward() call. The hang does not happen for smaller batch size or smaller model size.

## To Reproduce

I cannot share the code but what I did essentially is swap the text encoder in this repo (https://github.com/antoine77340/MIL-NCE_HowTo100M) to a BERT model from huggingface transformers. 


## Expected behavior
Doesn't hang

## Environment

 - PyTorch Version (e.g., 1.0): 1.7
 - OS (e.g., Linux): Ubuntu 18.04
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Python version: 3.8
 - CUDA/cuDNN version: 11.1
 - GPU models and configuration: 8 V100 GPUs
 - Any other relevant information: I am trying within a Docker Machine container built from NVIDIA's official images.

## Additional context
Here is the last few lines of NCCL debug output (too long to post all debug output). Seems to get stuck in AllReduce.

lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 18 sendbuff 0x7fc480707800 recvbuff 0x7fc480707800 count 7079424 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 16 sendbuff 0x7f605f104800 recvbuff 0x7f605f104800 count 7079424 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 19 sendbuff 0x7fc482209000 recvbuff 0x7fc482209000 count 7079424 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 17 sendbuff 0x7f6060c06000 recvbuff 0x7f6060c06000 count 7079424 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 1a sendbuff 0x7fc483d0a800 recvbuff 0x7fc483d0a800 count 7079424 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 1b sendbuff 0x7fc48580c000 recvbuff 0x7fc48580c000 count 7079424 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 18 sendbuff 0x7f6062707800 recvbuff 0x7f6062707800 count 7079424 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 1c sendbuff 0x7fc488000000 recvbuff 0x7fc488000000 count 7079040 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 19 sendbuff 0x7f6064209000 recvbuff 0x7f6064209000 count 7079424 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 1d sendbuff 0x7fc489b01200 recvbuff 0x7fc489b01200 count 24680448 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459429:3720265 [4] NCCL INFO AllReduce: opCount 1e sendbuff 0x7fc48f927200 recvbuff 0x7fc48f927200 count 12595200 datatype 7 op 0 root 0 comm 0x7fc5b0001060 [nranks=8] stream 0x56533e341eb0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 1a sendbuff 0x7f6065d0a800 recvbuff 0x7f6065d0a800 count 7079424 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 1b sendbuff 0x7f606780c000 recvbuff 0x7f606780c000 count 7079424 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 1c sendbuff 0x7f606a000000 recvbuff 0x7f606a000000 count 7079040 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 1d sendbuff 0x7f606bb01200 recvbuff 0x7f606bb01200 count 24680448 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0
lhrai88-lx:3459432:3720249 [7] NCCL INFO AllReduce: opCount 1e sendbuff 0x7f6071927200 recvbuff 0x7f6071927200 count 12595200 datatype 7 op 0 root 0 comm 0x7f619c001060 [nranks=8] stream 0x55fd88a504b0