Repository: pytorch
Issue: NCCL Network is unreachable / Connection refused when initializing DDP
Labels: oncall: distributed,module: nccl

Crossposting from https://discuss.pytorch.org/t/nccl-network-is-unreachable-connection-refused-when-initializing-ddp/137352, with some additional information added.

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
Hi, I’m trying to run a simple distributed PyTorch job across using GPU/NCCL across 2 `g4dn.xlarge` nodes. The process group seems to initialize fine, but when trying to wrap the model in DDP there is a NCCL connection error.

## To Reproduce

**Failure point:**

```
model = DistributedDataParallel(model, device_ids=[rank], output_device=rank)
```

**Environment:**
```
Torch: 1.9.0+cu111
NCCL: 2.7.8
```
**Logs with NCCL_DEBUG=INFO:**
```
Rank 0:
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO Bootstrap : Using [0]ens5:172.31.0.137<0> [1]veth75fa783:fe80::14ad:64ff:fe64:b31e%veth75fa783<0>
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
[0m 
[0m ip-172-31-0-137:14605:14673 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO NET/Socket : Using [0]ens5:172.31.0.137<0> [1]veth75fa783:fe80::14ad:64ff:fe64:b31e%veth75fa783<0>
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO Using network Socket
[0m NCCL version 2.7.8+cuda11.1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 00/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 01/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 02/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 03/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1|-1->0->1/-1/-1 [1] 1/-1/-1->0->-1|-1->0->1/-1/-1 [2] -1/-1/-1->0->1|1->0->-1/-1/-1 [3] -1/-1/-1->0->1|1->0->-1/-1/-1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 00 : 1[1e0] -> 0[1e0] [receive] via NET/Socket/0
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 00 : 0[1e0] -> 1[1e0] [send] via NET/Socket/0

Rank 1:
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO Bootstrap : Using [0]ens5:172.31.86.69<0> [1]veth0bd9843:fe80::6c37:83ff:fe11:cbd6%veth0bd9843<0>
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
[0m 
[0m ip-172-31-86-69:1538:1569 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO NET/Socket : Using [0]ens5:172.31.86.69<0> [1]veth0bd9843:fe80::6c37:83ff:fe11:cbd6%veth0bd9843<0>
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO Using network Socket
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Trees [0] -1/-1/-1->1->0|0->1->-1/-1/-1 [1] -1/-1/-1->1->0|0->1->-1/-1/-1 [2] 0/-1/-1->1->-1|-1->1->0/-1/-1 [3] 0/-1/-1->1->-1|-1->1->0/-1/-1
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 00 : 0[1e0] -> 1[1e0] [receive] via NET/Socket/0
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 00 : 1[1e0] -> 0[1e0] [send] via NET/Socket/0
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 01 : 0[1e0] -> 1[1e0] [receive] via NET/Socket/1
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 01 : 1[1e0] -> 0[1e0] [send] via NET/Socket/1
[0m 
[0m ip-172-31-86-69:1538:1572 [0] include/socket.h:403 NCCL WARN Connect to fe80::14ad:64ff:fe64:b31e%7<54043> failed : Network is unreachable
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO transport/net_socket.cc:313 -> 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO include/net.h:21 -> 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO transport/net.cc:161 -> 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO transport.cc:68 -> 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO init.cc:766 -> 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO init.cc:840 -> 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO group.cc:73 -> 2 [Async thread]

Rank 0:
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 01 : 1[1e0] -> 0[1e0] [receive] via NET/Socket/1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 01 : 0[1e0] -> 1[1e0] [send] via NET/Socket/1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m 
[0m ip-172-31-0-137:14605:14688 [0] include/socket.h:403 NCCL WARN Connect to 172.31.86.69<37817> failed : Connection refused
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO bootstrap.cc:95 -> 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO bootstrap.cc:363 -> 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO transport.cc:59 -> 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO init.cc:766 -> 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO init.cc:840 -> 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO group.cc:73 -> 2 [Async thread]
```
**Final Output:**
```
  File "/home/ray/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 496, in __init__
    dist._verify_model_across_ranks(self.process_group, parameters)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8
ncclSystemError: System call (socket, malloc, munmap, etc) failed.                                                                                                                                        
```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
The script should successfully run without hanging/erroring out.

## Environment

```
$ python collect_env.py

Collecting environment information...
PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.10

Python version: 3.7.7 (default, May  7 2020, 21:25:33)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.3.0-1032-aws-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 450.51.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] botorch==0.5.0
[pip3] gpytorch==1.5.1
[pip3] numpy==1.19.5
[pip3] pytorch-lightning==1.4.9
[pip3] torch==1.9.0+cu111
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.10.0+cu111
[conda] botorch                   0.5.0                    pypi_0    pypi
[conda] gpytorch                  1.5.1                    pypi_0    pypi
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] pytorch-lightning         1.4.9                    pypi_0    pypi
[conda] torch                     1.9.0+cu111              pypi_0    pypi
[conda] torchmetrics              0.6.0                    pypi_0    pypi
[conda] torchvision               0.10.0+cu111             pypi_0    pypi
```
```
$ ifconfig

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:eeff:fe69:242a  prefixlen 64  scopeid 0x20<link>
        ether 02:42:ee:69:24:2a  txqueuelen 0  (Ethernet)
        RX packets 380  bytes 1492631 (1.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 386  bytes 32576 (32.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.19.81  netmask 255.255.224.0  broadcast 172.31.31.255
        inet6 fe80::46e:36ff:fee3:8e6b  prefixlen 64  scopeid 0x20<link>
        ether 06:6e:36:e3:8e:6b  txqueuelen 1000  (Ethernet)
        RX packets 5422625  bytes 7946170910 (7.9 GB)
        RX errors 0  dropped 757  overruns 0  frame 0
        TX packets 402722  bytes 26773108 (26.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 295188  bytes 5904985733 (5.9 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 295188  bytes 5904985733 (5.9 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth05d2fa2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::b486:8bff:fe62:2182  prefixlen 64  scopeid 0x20<link>
        ether b6:86:8b:62:21:82  txqueuelen 0  (Ethernet)
        RX packets 380  bytes 1497951 (1.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 393  bytes 32986 (32.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```

## Additional context

<!-- Add any other context about the problem here. -->

A few other things worth mentioning:

1. The script runs successfully if I hardcode `os.environ["NCCL_SOCKET_IFNAME"]="ens5"`. Is there some incompatibility with `veth*`?
2. The script runs when downgrading to Torch 1.6.0 with CUDA 10.2 and NCCL 2.4.8. Note from the NCCL logs below that both `ens5` and `veth*` are still being used.
```
Rank 0:
ip-172-31-92-218:4760:4826 [0] NCCL INFO Bootstrap : Using [0]ens5:172.31.92.218<0> [1]vethe2e6725:fe80::4811:8aff:fe89:315f%vethe2e6725<0>
ip-172-31-92-218:4760:4826 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

ip-172-31-92-218:4760:4826 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ip-172-31-92-218:4760:4826 [0] NCCL INFO NET/Socket : Using [0]ens5:172.31.92.218<0> [1]vethe2e6725:fe80::4811:8aff:fe89:315f%vethe2e6725<0>
NCCL version 2.4.8+cuda10.2
ip-172-31-92-218:4760:4837 [0] NCCL INFO Setting affinity for GPU 0 to 0f
ip-172-31-92-218:4760:4837 [0] NCCL INFO Could not find real path of /sys/class/net/vethe2e6725/device
ip-172-31-92-218:4760:4837 [0] NCCL INFO include/net.h:19 -> 2
ip-172-31-92-218:4760:4837 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance :  PHB SYS
ip-172-31-92-218:4760:4837 [0] NCCL INFO Channel 00 :    0   1
ip-172-31-92-218:4760:4837 [0] NCCL INFO Ring 00 : 1 -> 0 [receive] via NET/Socket/0
ip-172-31-92-218:4760:4837 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread
ip-172-31-92-218:4760:4837 [0] NCCL INFO Ring 00 : 0 -> 1 [send] via NET/Socket/0
ip-172-31-92-218:4760:4837 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees disabled
ip-172-31-92-218:4760:4837 [0] NCCL INFO comm 0x7f0a78002440 rank 0 nranks 2 cudaDev 0 nvmlDev 0 - Init COMPLETE
ip-172-31-92-218:4760:4826 [0] NCCL INFO Launch mode Parallel

Rank 1:
ip-172-31-75-36:589:618 [0] NCCL INFO Bootstrap : Using [0]ens5:172.31.75.36<0> [1]veth8c98668:fe80::a041:e8ff:fe11:1330%veth8c98668<0>
ip-172-31-75-36:589:618 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

ip-172-31-75-36:589:618 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ip-172-31-75-36:589:618 [0] NCCL INFO NET/Socket : Using [0]ens5:172.31.75.36<0> [1]veth8c98668:fe80::a041:e8ff:fe11:1330%veth8c98668<0>
ip-172-31-75-36:589:622 [0] NCCL INFO Setting affinity for GPU 0 to 0f
ip-172-31-75-36:589:622 [0] NCCL INFO Could not find real path of /sys/class/net/veth8c98668/device
ip-172-31-75-36:589:622 [0] NCCL INFO include/net.h:19 -> 2
ip-172-31-75-36:589:622 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance :  PHB SYS
ip-172-31-75-36:589:622 [0] NCCL INFO Ring 00 : 0 -> 1 [receive] via NET/Socket/0
ip-172-31-75-36:589:622 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread
ip-172-31-75-36:589:622 [0] NCCL INFO Ring 00 : 1 -> 0 [send] via NET/Socket/0
ip-172-31-75-36:589:622 [0] NCCL INFO comm 0x7f0400002440 rank 1 nranks 2 cudaDev 0 nvmlDev 0 - Init COMPLETE
```



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang