Repository: pytorch
Issue: PyTorch gets stuck when using an NVLink/A6000 and more than two GPUs
Labels: oncall: distributed,triaged,module: ddp

### 🐛 Describe the bug

When I utilize PyTorch’s distributed data parallel (DDP) to train the [ImageNet example](https://github.com/pytorch/examples/tree/main/imagenet) with two GPUs, NVLink is used successfully based on the performance counters. However, as soon as I increase the number of GPUs to three or more (up to eight), the training loop gets stuck at the very beginning.

If I set the environment variable `NCCL_P2P_DISABLE=1`, I can use as many GPUs as I like, but I obviously don’t get the benefits of NVLink.

Output with two GPUs:

```
Use GPU: 0 for training
Use GPU: 1 for training
=> creating model 'resnet50'
=> creating model 'resnet50'
Epoch: [0][   1/5005]   Time  6.435 ( 6.435)    Data  4.396 ( 4.396)    Loss 7.1897e+00 (7.1897e+00)    Acc@1   0.00 (  0.00)   Acc@5   0.00 (  0.00)
Epoch: [0][   1/5005]   Time  6.439 ( 6.439)    Data  4.302 ( 4.302)    Loss 7.0490e+00 (7.0490e+00)    Acc@1   0.00 (  0.00)   Acc@5   2.34 (  2.34)
Epoch: [0][  11/5005]   Time  0.303 ( 0.854)    Data  0.000 ( 0.400)    Loss 8.3196e+00 (7.8798e+00)    Acc@1   0.00 (  0.07)   Acc@5   0.00 (  0.28)
Epoch: [0][  11/5005]   Time  0.303 ( 0.853)    Data  0.000 ( 0.391)    Loss 8.3118e+00 (8.1050e+00)    Acc@1   0.00 (  0.07)   Acc@5   0.78 (  1.07)
Epoch: [0][  21/5005]   Time  0.323 ( 0.592)    Data  0.000 ( 0.214)    Loss 7.3862e+00 (8.0053e+00)    Acc@1   0.00 (  0.04)   Acc@5   0.00 (  0.22)
Epoch: [0][  21/5005]   Time  0.324 ( 0.591)    Data  0.113 ( 0.215)    Loss 8.0911e+00 (8.1161e+00)    Acc@1   0.78 (  0.11)   Acc@5   0.78 (  0.78)
```

Output with eight GPUs

```
Use GPU: 0 for training
Use GPU: 4 for training
Use GPU: 5 for training
Use GPU: 7 for training
Use GPU: 3 for training
Use GPU: 1 for training
Use GPU: 2 for training
Use GPU: 6 for training
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
=> creating model 'resnet50'
```

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.17

Python version: 3.7.10 (default, Jun  4 2021, 14:48:32)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-113-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 11.3.109
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000
GPU 4: NVIDIA RTX A6000
GPU 5: NVIDIA RTX A6000
GPU 6: NVIDIA RTX A6000
GPU 7: NVIDIA RTX A6000

Nvidia driver version: 510.73.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchfile==0.1.0
[pip3] torchnet==0.0.4
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] cudatoolkit-dev           11.3.1           py37h5e8e339_0    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py37h7f8727e_0
[conda] mkl_fft                   1.3.1            py37hd3c417c_0
[conda] mkl_random                1.2.2            py37h51133e4_0
[conda] numpy                     1.20.2                   pypi_0    pypi
[conda] numpy-base                1.21.5           py37ha15fc14_3
[conda] pytorch                   1.11.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0               py37_cu113    pytorch
[conda] torchvision               0.12.0               py37_cu113    pytorch

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501