Repository: pytorch
Issue: dist.init_process_group('nccl', ...) hangs in some combinations of pytorch+python+cuda version
Labels: oncall: distributed,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

dist.init_process_group('nccl') hangs on some version of pytorch+python+cuda version

## To Reproduce

Steps to reproduce the behavior:

1. conda create -n py38 python=3.8
2. conda activate py38
1. conda install pytorch torchvision torchaudio cudatoolkit=10.1 -c pytorch
1. open python3
1. run code:
```python3
import os, torch.distributed as dist
os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '55408'
dist.init_process_group(backend='nccl', world_size=1, rank=0)
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
It should not hang. If you change backend='gloo', it will not hang. But if you use nccl backend, it will always hang.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration:
GPU 0: A100-SXM4-40GB

Nvidia driver version: 450.51.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] numpydoc==1.1.0
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl    https://repo.anaconda.com/pkgs/main
[conda] cudatoolkit               10.1.243             h6bb024c_0    defaults
[conda] mkl                       2020.2                      256    https://repo.anaconda.com/pkgs/main
[conda] mkl-service               2.3.0            py38he904b0f_0    https://repo.anaconda.com/pkgs/main
[conda] mkl_fft                   1.2.0            py38h23d657b_0    https://repo.anaconda.com/pkgs/main
[conda] mkl_random                1.1.1            py38h0573a6f_0    https://repo.anaconda.com/pkgs/main
[conda] numpy                     1.19.2           py38h54aff64_0    https://repo.anaconda.com/pkgs/main
[conda] numpy-base                1.19.2           py38hfa32c7d_0    https://repo.anaconda.com/pkgs/main
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1    https://repo.anaconda.com/pkgs/main
[conda] pytorch                   1.7.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                py38_cu101    pytorch


## Additional context

<!-- Add any other context about the problem here. -->
cc: @teng-li 

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @pietern @agolynski @SciPioneer @mrzzd @cbalioglu