Repository: pytorch
Issue: all reduce hangs and does not throw exception when CUDA_VISIBLE_DEVICES is not set properly using the NCCL backend
Labels: oncall: distributed,module: nccl

## 🐛 Bug

Running an allreduce on CUDA tensors across multiple machines when using GPU 0 only on each machine causes a hang with the NCCL backend and works fine with gloo.

## To Reproduce

Steps to reproduce the behavior:

1.  Have at least 2 machines, and an mpi hosts file with both machines
1. `mpiexec -l -f hosts-file -ppn 1 python script.py gloo` works fine
1. `mpiexec -l -f hosts-file -ppn 1 python script.py nccl` hangs after "all reduce tensor"

It seems to work fine for NCCL if you use `-ppn 8` or if you use PyTorch 1.5.

```py
import torch
import socket
from mpi4py import MPI
import os
import torch.distributed as dist
import sys


def main():
    backend = sys.argv[1]
    shared_comm = MPI.COMM_WORLD.Split_type(MPI.COMM_TYPE_SHARED)
    device_index = shared_comm.rank
    # device_index = MPI.COMM_WORLD.rank

    device = torch.device("cuda", index=device_index)
    torch.cuda.set_device(device)
    hostname = socket.gethostbyname(socket.getfqdn())
    os.environ["MASTER_ADDR"] = MPI.COMM_WORLD.bcast(hostname, root=0)
    os.environ["RANK"] = str(MPI.COMM_WORLD.rank)
    os.environ["WORLD_SIZE"] = str(MPI.COMM_WORLD.size)
    os.environ["MASTER_PORT"] = "20000"
    dist.init_process_group(init_method="env://", backend=backend)

    print(f"all reduce tensor hostname={socket.gethostbyname(socket.getfqdn())} device={device}")
    t = torch.zeros((2,2), device=device)
    torch.distributed.all_reduce(t)
    torch.cuda.synchronize()
    print("all reduce complete")

if __name__ == "__main__":
    main()
```

gloo output:
```
[3] all reduce tensor hostname=10.100.65.118 device=cuda:0
[0] all reduce tensor hostname=10.100.53.129 device=cuda:0
[2] all reduce tensor hostname=10.100.120.96 device=cuda:0
[1] all reduce tensor hostname=10.100.128.69 device=cuda:0
[3] all reduce complete
[0] all reduce complete
[1] all reduce complete
[2] all reduce complete
```

nccl output:
```
[0] NCCL version 2.7.8+cuda11.0
[0] all reduce tensor hostname=10.100.53.129 device=cuda:0
[1] all reduce tensor hostname=10.100.128.69 device=cuda:0
[3] all reduce tensor hostname=10.100.65.118 device=cuda:0
[2] all reduce tensor hostname=10.100.120.96 device=cuda:0
```

## Expected behavior

Seems like it should not hang, at least produce an error if I am doing something wrong.

## Environment

PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.11.1

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.1.74
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
GPU 4: Tesla V100-SXM2-32GB
GPU 5: Tesla V100-SXM2-32GB
GPU 6: Tesla V100-SXM2-32GB
GPU 7: Tesla V100-SXM2-32GB

Nvidia driver version: 450.102.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] pytorch-memlab==0.2.3
[pip3] torch==1.7.1+cu110
[pip3] torch-bst==0.0.0
[pip3] torch-streams==0.0.1
[pip3] torchvision==0.8.2
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] pytorch-memlab            0.2.3                    pypi_0    pypi
[conda] torch                     1.7.1+cu110              pypi_0    pypi
[conda] torch-bst                 0.0.0                    pypi_0    pypi
[conda] torch-streams             0.0.1                     dev_0    <develop>
[conda] torchvision               0.8.2                    pypi_0    pypi
## Additional context

<!-- Add any other context about the problem here. -->

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu