Repository: pytorch
Issue: RuntimeError: Connection reset by peer when backened by NCCL
Labels: module: cuda,triaged,module: nccl

### 🐛 Describe the bug


I run the following script named "index.py" on 2 machines with CUDA GPU installed.


```python
import datetime
import os
import torch.distributed as dist
import sys
backend='nccl'
rank=int(sys.argv[1])
ifnames=["enp9s0",'eno1']
os.environ['GLOO_SOCKET_IFNAME']=ifnames[rank]
os.environ['NCCL_SOCKET_IFNAME']=ifnames[rank]
dist.init_process_group(backend=backend, rank=rank,init_method='tcp://192.168.1.2:23456',  world_size=2,timeout=datetime.timedelta(seconds=15))
print("ok")
````

on Machine A (master, with a GTX 1650):
```
python index.py 0
```

on Machine B(with a RTX 3060):
```
python index.py 1
```

Machine A prints 'ok' as expected.

But Machine B prints the following error:

```
Traceback (most recent call last):
  File "/mnt/sdc2/file/project/distributed/index.py", line 10, in <module>
    dist.init_process_group(backend=backend, rank=rank,init_method='tcp://192.168.1.2:23456',  world_size=2,timeout=datetime.timedelta(seconds=15))
  File "/root/miniconda3/envs/vision/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 611, in init_process_group
    default_pg._set_sequence_number_for_group()
RuntimeError: Connection reset by peer
```

If I changed backend into 'gloo' , then both Machine A and Machine B prints 'ok' as expected.

Any ideas on what is going wrong when backended by 'nccl' ? 


### Versions

PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.5.50
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    defaults
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640    defaults
[conda] mkl-service               2.4.0            py39h7f8727e_0    defaults
[conda] mkl_fft                   1.3.1            py39hd3c417c_0    defaults
[conda] mkl_random                1.2.2            py39h51133e4_0    defaults
[conda] numpy                     1.21.2           py39h20f2e39_0    defaults
[conda] numpy-base                1.21.2           py39h79a1101_0    defaults
[conda] pytorch                   1.10.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.0               py39_cu113    pytorch
[conda] torchvision               0.11.1               py39_cu113    pytorch


cc @ngimel