Repository: pytorch
Issue: nccl deadlock
Labels: oncall: distributed,triaged

### 🐛 Describe the bug

Using `torch.distributed.barrier()` doesn't work with NCCL. I use the code
```python
import torch

def train() -> None:

    torch.distributed.init_process_group('nccl')

    local_rank = torch.distributed.get_rank()
    torch.cuda.set_device(local_rank)
    torch.tensor([1]).to(local_rank)

    print("START", torch.distributed.get_rank())
    torch.distributed.barrier()
    print("GOOD", torch.distributed.get_rank())

    torch.distributed.destroy_process_group()


if __name__ == "__main__":
    train()
```
and I run it with
```bash
torchrun \
--standalone \
--nnodes 1 \
--nproc-per-node 4 \
--rdzv-backend=c10d \
--rdzv-endpoint=localhost:0 \
test_train.py
```
The workers don't successfully pass through the barrier. Playing with recent versions of torch, I get slightly different, but always unsuccessful results:
- All workers print `START`, all workers but rank 0 pass through and print `GOOD`
- All workers print `START`, no workers pass through and print `GOOD`

Changing to 2 GPUs doesn't make any difference.

**When I change the backend to `gloo`, the program completes successfully every time.**

Here is the output when I run with `NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=ALL TORCH_DISTRIBUTED_DEBUG=INFO` https://gist.github.com/nikonikolov/0864be35988b8cddc2b5fd30baa0d731

### Versions
```
Collecting environment information...                                                                                                                                                                                     15:42:58 [25/1819]
PyTorch version: 2.3.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux trixie/sid (x86_64)
GCC version: (Debian 13.3.0-1) 13.3.0
Clang version: Could not collect
CMake version: version 3.29.6
Libc version: glibc-2.38

Python version: 3.11.9 (main, Apr 10 2024, 13:16:36) [GCC 13.2.0] (64-bit runtime)
Python platform: Linux-6.7.12-amd64-x86_64-with-glibc2.38
Is CUDA available: True
CUDA runtime version: 12.1.66
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000

Nvidia driver version: 555.42.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               96
On-line CPU(s) list:                  0-95
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 74F3 24-Core Processor
CPU family:                           25
Model:                                1
Thread(s) per core:                   2
Core(s) per socket:                   24
Socket(s):                            2
Stepping:                             1
Frequency boost:                      enabled
CPU(s) scaling MHz:                   61%
CPU max MHz:                          4037.5000
CPU min MHz:                          1500.0000
BogoMIPS:                             6400.18
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_ap
icid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_co
re perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_oc
cup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku o
spke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
Virtualization:                       AMD-V
L1d cache:                            1.5 MiB (48 instances)
L1i cache:                            1.5 MiB (48 instances)
L2 cache:                             24 MiB (48 instances)
L3 cache:                             512 MiB (16 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-23,48-71
NUMA node1 CPU(s):                    24-47,72-95
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; Safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.3.1+cu118
[pip3] triton==2.3.1
[conda] Could not collect
```

This was also tested with all combinations of cuda 11.8, cuda 12.1, torch==2.2.0, torch==2.3.0, python3.11, python3.10

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @chauhang @d4l3k