Repository: pytorch
Issue: torch.distributed hangs at barrier()
Labels: oncall: distributed,module: c10d

### 🐛 Describe the bug

I am unable to train a huggingface model using torch.distributed on multiple GPUs in a single node even though single GPU training works. I created an [issue](https://github.com/huggingface/transformers/issues/15618) with huggingface and was directed to the [torch-distributed-gpu-test.py](https://github.com/stas00/toolbox/blob/master/pytorch/torch-distributed-gpu-test.py) diagnostic script which hangs for me at the second `barrier()` call. However, it's not clear if it's an issue with our GPUs setup or an issue with my pytorch installation in particular (since others in my group are able to use torch.distributed) and in either case, how to resolve it. 

Here are the logs for the diagnostic script run with different logging settings:

```NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py```: [log0.txt](https://github.com/huggingface/transformers/files/8055103/log0.txt)

```NCCL_DEBUG_SUBSYS=NET NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py```: [log1.txt](https://github.com/huggingface/transformers/files/8055099/log1.txt)

```NCCL_DEBUG_SUBSYS=ALL NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py```: [log2.txt](https://github.com/huggingface/transformers/files/8055097/log2.txt) 

<details><summary>

`NCCL_SOCKET_IFNAME=lo NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py`
</summary>

```
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
ava-s0:537553:537553 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537553:537553 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537553:537553 [0] NCCL INFO NET/IB : No device found.
ava-s0:537553:537553 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537553:537553 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
ava-s0:537559:537559 [6] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537559:537559 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537559:537559 [6] NCCL INFO NET/IB : No device found.
ava-s0:537559:537559 [6] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537559:537559 [6] NCCL INFO Using network Socket
ava-s0:537556:537556 [3] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537556:537556 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537556:537556 [3] NCCL INFO NET/IB : No device found.
ava-s0:537556:537556 [3] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537556:537556 [3] NCCL INFO Using network Socket
ava-s0:537557:537557 [4] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537557:537557 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537557:537557 [4] NCCL INFO NET/IB : No device found.
ava-s0:537557:537557 [4] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537557:537557 [4] NCCL INFO Using network Socket
ava-s0:537555:537555 [2] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537555:537555 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537555:537555 [2] NCCL INFO NET/IB : No device found.
ava-s0:537555:537555 [2] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537555:537555 [2] NCCL INFO Using network Socket
ava-s0:537554:537554 [1] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537554:537554 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537554:537554 [1] NCCL INFO NET/IB : No device found.
ava-s0:537554:537554 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537554:537554 [1] NCCL INFO Using network Socket
ava-s0:537560:537560 [7] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537560:537560 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537560:537560 [7] NCCL INFO NET/IB : No device found.
ava-s0:537560:537560 [7] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537560:537560 [7] NCCL INFO Using network Socket
ava-s0:537558:537558 [5] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537558:537558 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537558:537558 [5] NCCL INFO NET/IB : No device found.
ava-s0:537558:537558 [5] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537558:537558 [5] NCCL INFO Using network Socket
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537558:537635 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4
ava-s0:537558:537635 [5] NCCL INFO Setting affinity for GPU 5 to ffff0000,ffff0000
ava-s0:537559:537623 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5
ava-s0:537559:537623 [6] NCCL INFO Setting affinity for GPU 6 to ffff0000,ffff0000
ava-s0:537560:537628 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6
ava-s0:537560:537628 [7] NCCL INFO Setting affinity for GPU 7 to ffff0000,ffff0000
ava-s0:537553:537622 [0] NCCL INFO Channel 00/02 :    0   2   3   4   5   7   6   1
ava-s0:537554:537627 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
ava-s0:537555:537626 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
ava-s0:537553:537622 [0] NCCL INFO Channel 01/02 :    0   2   3   4   5   7   6   1
ava-s0:537554:537627 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
ava-s0:537556:537624 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2
ava-s0:537555:537626 [2] NCCL INFO Setting affinity for GPU 2 to ffff,0000ffff
ava-s0:537557:537625 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3
ava-s0:537553:537622 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
ava-s0:537556:537624 [3] NCCL INFO Setting affinity for GPU 3 to ffff,0000ffff
ava-s0:537557:537625 [4] NCCL INFO Setting affinity for GPU 4 to ffff0000,ffff0000
ava-s0:537553:537622 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537559:537623 [6] NCCL INFO Channel 00 : 6[b1000] -> 1[1b000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537553:537622 [0] NCCL INFO Channel 00 : 0[1a000] -> 2[3d000] via P2P/IPC
ava-s0:537555:537626 [2] NCCL INFO Channel 00 : 2[3d000] -> 3[3e000] via direct shared memory
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537559:537623 [6] NCCL INFO Channel 01 : 6[b1000] -> 1[1b000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Channel 01 : 0[1a000] -> 2[3d000] via P2P/IPC
ava-s0:537555:537626 [2] NCCL INFO Channel 01 : 2[3d000] -> 3[3e000] via direct shared memory
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Channel 00 : 4[88000] -> 5[89000] via direct shared memory
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Channel 01 : 4[88000] -> 5[89000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Channel 00 : 5[89000] -> 7[b2000] via P2P/IPC
ava-s0:537554:537627 [1] NCCL INFO Channel 00 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Channel 01 : 5[89000] -> 7[b2000] via P2P/IPC
ava-s0:537554:537627 [1] NCCL INFO Channel 01 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Channel 00 : 3[3e000] -> 4[88000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537556:537624 [3] NCCL INFO Channel 01 : 3[3e000] -> 4[88000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Connected all rings
ava-s0:537560:537628 [7] NCCL INFO Channel 00 : 7[b2000] -> 6[b1000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Channel 00 : 5[89000] -> 6[b1000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Channel 01 : 7[b2000] -> 6[b1000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Channel 01 : 5[89000] -> 6[b1000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537557:537625 [4] NCCL INFO Connected all rings
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537555:537626 [2] NCCL INFO Connected all rings
ava-s0:537559:537623 [6] NCCL INFO Connected all rings
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537560:537628 [7] NCCL INFO Connected all rings
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537557:537625 [4] NCCL INFO Channel 00 : 4[88000] -> 3[3e000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Channel 00 : 6[b1000] -> 7[b2000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537554:537627 [1] NCCL INFO Connected all rings
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537557:537625 [4] NCCL INFO Channel 01 : 4[88000] -> 3[3e000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Connected all rings
ava-s0:537559:537623 [6] NCCL INFO Channel 01 : 6[b1000] -> 7[b2000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Connected all trees
ava-s0:537560:537628 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537560:537628 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537553:537622 [0] NCCL INFO Connected all rings
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Channel 00 : 6[b1000] -> 5[89000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Channel 01 : 6[b1000] -> 5[89000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Channel 00 : 5[89000] -> 4[88000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Channel 01 : 5[89000] -> 4[88000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Connected all trees
ava-s0:537558:537635 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537558:537635 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537559:537623 [6] NCCL INFO Connected all trees
ava-s0:537559:537623 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537559:537623 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Channel 00 : 3[3e000] -> 2[3d000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Channel 01 : 3[3e000] -> 2[3d000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Channel 00 : 1[1b000] -> 2[3d000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Channel 01 : 1[1b000] -> 2[3d000] via direct shared memory
ava-s0:537557:537625 [4] NCCL INFO Connected all trees
ava-s0:537557:537625 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537557:537625 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537553:537622 [0] NCCL INFO Connected all trees
ava-s0:537553:537622 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537553:537622 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537555:537626 [2] NCCL INFO Channel 00 : 2[3d000] -> 1[1b000] via direct shared memory
ava-s0:537555:537626 [2] NCCL INFO Channel 01 : 2[3d000] -> 1[1b000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Connected all trees
ava-s0:537556:537624 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537556:537624 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537554:537627 [1] NCCL INFO Connected all trees
ava-s0:537554:537627 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537554:537627 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537555:537626 [2] NCCL INFO Connected all trees
ava-s0:537555:537626 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537555:537626 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537553:537622 [0] NCCL INFO comm 0x7f3450002fb0 rank 0 nranks 8 cudaDev 0 busId 1a000 - Init COMPLETE
ava-s0:537557:537625 [4] NCCL INFO comm 0x7f2bc0002fb0 rank 4 nranks 8 cudaDev 4 busId 88000 - Init COMPLETE
ava-s0:537559:537623 [6] NCCL INFO comm 0x7fbe4c002fb0 rank 6 nranks 8 cudaDev 6 busId b1000 - Init COMPLETE
ava-s0:537553:537553 [0] NCCL INFO Launch mode Parallel
ava-s0:537556:537624 [3] NCCL INFO comm 0x7f75d0002fb0 rank 3 nranks 8 cudaDev 3 busId 3e000 - Init COMPLETE
ava-s0:537554:537627 [1] NCCL INFO comm 0x7f40ec002fb0 rank 1 nranks 8 cudaDev 1 busId 1b000 - Init COMPLETE
ava-s0:537558:537635 [5] NCCL INFO comm 0x7f5688002fb0 rank 5 nranks 8 cudaDev 5 busId 89000 - Init COMPLETE
ava-s0:537555:537626 [2] NCCL INFO comm 0x7faf08002fb0 rank 2 nranks 8 cudaDev 2 busId 3d000 - Init COMPLETE
ava-s0:537560:537628 [7] NCCL INFO comm 0x7f9f28002fb0 rank 7 nranks 8 cudaDev 7 busId b2000 - Init COMPLETE
[ava-s0.ics.uci.edu-0] is OK (global rank: 0/8)
```
</details>

<details><summary>

`NCCL_SOCKET_IFNAME=lo NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py`

</summary>

```
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
ava-s0:538447:538447 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:538447:538447 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538447:538447 [0] NCCL INFO NET/IB : No device found.
ava-s0:538447:538447 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:538447:538447 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
ava-s0:538448:538448 [1] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:538448:538448 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538448:538448 [1] NCCL INFO NET/IB : No device found.
ava-s0:538448:538448 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:538448:538448 [1] NCCL INFO Using network Socket
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538448:538500 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
ava-s0:538448:538500 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
ava-s0:538447:538499 [0] NCCL INFO Channel 00/02 :    0   1
ava-s0:538447:538499 [0] NCCL INFO Channel 01/02 :    0   1
ava-s0:538447:538499 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
ava-s0:538447:538499 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Channel 00 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Channel 01 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538448:538500 [1] NCCL INFO Connected all rings
ava-s0:538448:538500 [1] NCCL INFO Connected all trees
ava-s0:538448:538500 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538448:538500 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538447:538499 [0] NCCL INFO Connected all rings
ava-s0:538447:538499 [0] NCCL INFO Connected all trees
ava-s0:538447:538499 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538447:538499 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538447:538499 [0] NCCL INFO comm 0x7f4afc002fb0 rank 0 nranks 2 cudaDev 0 busId 1a000 - Init COMPLETE
ava-s0:538447:538447 [0] NCCL INFO Launch mode Parallel
ava-s0:538448:538500 [1] NCCL INFO comm 0x7f16ac002fb0 rank 1 nranks 2 cudaDev 1 busId 1b000 - Init COMPLETE
[ava-s0.ics.uci.edu-1] is OK (global rank: 1/2)
```
</details>


<details><summary>

`NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py`
</summary>

```
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
ava-s0:538740:538740 [0] NCCL INFO Bootstrap : Using eno0:128.195.10.163<0>
ava-s0:538740:538740 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538740:538740 [0] NCCL INFO NET/IB : No device found.
ava-s0:538740:538740 [0] NCCL INFO NET/Socket : Using [0]eno0:128.195.10.163<0>
ava-s0:538740:538740 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
ava-s0:538741:538741 [1] NCCL INFO Bootstrap : Using eno0:128.195.10.163<0>
ava-s0:538741:538741 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538741:538741 [1] NCCL INFO NET/IB : No device found.
ava-s0:538741:538741 [1] NCCL INFO NET/Socket : Using [0]eno0:128.195.10.163<0>
ava-s0:538741:538741 [1] NCCL INFO Using network Socket
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Channel 00/02 :    0   1
ava-s0:538740:538760 [0] NCCL INFO Channel 01/02 :    0   1
ava-s0:538741:538761 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
ava-s0:538740:538760 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
ava-s0:538741:538761 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
ava-s0:538740:538760 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Channel 00 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Channel 01 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538740:538760 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538741:538761 [1] NCCL INFO Connected all rings
ava-s0:538741:538761 [1] NCCL INFO Connected all trees
ava-s0:538741:538761 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538741:538761 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538740:538760 [0] NCCL INFO Connected all rings
ava-s0:538740:538760 [0] NCCL INFO Connected all trees
ava-s0:538740:538760 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538740:538760 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538741:538761 [1] NCCL INFO comm 0x7f5690002fb0 rank 1 nranks 2 cudaDev 1 busId 1b000 - Init COMPLETE
ava-s0:538740:538760 [0] NCCL INFO comm 0x7fc454002fb0 rank 0 nranks 2 cudaDev 0 busId 1a000 - Init COMPLETE
ava-s0:538740:538740 [0] NCCL INFO Launch mode Parallel
[ava-s0.ics.uci.edu-0] is OK (global rank: 0/2)
```
</details>

Some of our observations were the many messages of the form: `Could not enable P2P between dev A and dev B` and the fact that different gpus seem to succeed in different runs.


### Versions

```
Collecting environment information...
PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.5 (default, Jun  4 2021, 12:28:51)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX
GPU 3: TITAN RTX
GPU 4: TITAN RTX
GPU 5: TITAN RTX
GPU 6: TITAN RTX
GPU 7: TITAN RTX

Nvidia driver version: 460.91.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.2.2
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.2.2
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.2.2
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.2.2
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.2.2
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.2.2
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.2.2
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.10.2
[pip3] torchaudio==0.10.2
[pip3] torchinfo==1.5.3
[pip3] torchtext==0.11.2
[pip3] torchvision==0.11.3
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h2bc3f7f_2
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0                     9_mkl    conda-forge
[conda] libcblas                  3.9.0                     9_mkl    conda-forge
[conda] mkl                       2021.3.0           h726a3e6_557    conda-forge
[conda] mkl-service               2.4.0            py39h3811e60_0    conda-forge
[conda] mkl_fft                   1.3.0            py39h42c9631_2
[conda] mkl_random                1.2.2            py39hde0f152_0    conda-forge
[conda] numpy                     1.20.3           py39hf144106_0
[conda] numpy-base                1.20.3           py39h74d4b33_0
[conda] pytorch                   1.10.2          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.2               py39_cu113    pytorch
[conda] torchinfo                 1.5.3              pyhd8ed1ab_0    conda-forge
[conda] torchtext                 0.11.2                     py39    pytorch
[conda] torchvision               0.11.3               py39_cu113    pytorch
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang