Repository: pytorch
Issue: P2p operations hang when mix the usage of default and non-default communication groups
Labels: oncall: distributed

### 🐛 Describe the bug

## Bug Description

**Issue**: Hang in `isend/irecv` operations when mixing default and non-default communication groups.

**Details**: Asynchronous communication operations (`isend/irecv`) exhibit a hanging issue when alternating between the default communication group and user-defined groups (`grp_0` and `grp_1`). The first set of operations using the default group succeeds, but subsequent operations using `grp_0` and `grp_1` hang.

## Reproduction Steps

1. Initialize three communication groups:
   - **Default group**: Ranks 0-5
   - **grp_0**: Ranks 0, 2, 4
   - **grp_1**: Ranks 1, 3, 5
2. Perform `isend/irecv` operations:
   - Rank 0 to Rank 2 (default group) - succeeds
   - Rank 1 to Rank 3 (default group) - succeeds
   - Rank 0 to Rank 2 (grp_0) - hangs
   - Rank 1 to Rank 3 (grp_1) - hangs

## Expected Behavior

Operations using both default and non-default communication groups should complete successfully without hanging.

## Actual Behavior

Operations hang when switching from the default group to `grp_0` or `grp_1` for the second set of `isend/irecv`.

## Additional Information

This bug only triggers when the world size is larger than four, using the below script.

```
# Pytorch script
import os
import torch
import torch.distributed as dist

NODE_RANK = int(os.getenv("RANK", '0'))
WORLD_SIZE = int(os.environ.get("WORLD_SIZE"))
MASTER_ADDR = os.environ.get("MASTER_ADDR")
MASTER_PORT = os.environ.get("MASTER_PORT")

class Bug:


    def __init__(self):

        dist.init_process_group(
            backend = "nccl",
            init_method = f"tcp://{MASTER_ADDR}:{MASTER_PORT}",
            rank = NODE_RANK,
            world_size = WORLD_SIZE,
        )

        self.rank = dist.get_rank()
        device_count = torch.cuda.device_count()
        device = self.rank % device_count
        torch.cuda.set_device(device)

        grp_0 = dist.new_group(range(0, WORLD_SIZE, 2), tag=f"grp{self.rank}_0")
        grp_1 = dist.new_group(range(1, WORLD_SIZE, 2), tag=f"grp{self.rank}_1")
        self.grp = grp_0 if self.rank in range(0, WORLD_SIZE, 2) else grp_1


    def run(self):
        
        tensor_recv_shape = torch.empty(
            (3), device=torch.cuda.current_device(), dtype=torch.int64
        )
        tensor_send_shape = torch.ones(
            (3), device=torch.cuda.current_device(), dtype=torch.int64
        )

        tensor_recv = torch.empty(
            (1024), device=torch.cuda.current_device(), dtype=torch.int64
        )
        tensor_send = torch.ones(
            (1024), device=torch.cuda.current_device(), dtype=torch.int64
        )

        if self.rank in [0, 1]:
            op = dist.P2POp(dist.isend, tensor_send_shape, self.rank+2)
            reqs = dist.batch_isend_irecv([op])
            for req in reqs:
                req.wait()
            torch.cuda.synchronize()
            print(f"rank {self.rank} isend default group done")
            
            print(f"rank {self.rank} isend non-default group begin")
            op = dist.P2POp(dist.isend, tensor_send, self.rank+2, self.grp)
            reqs = dist.batch_isend_irecv([op])
            for req in reqs:
                req.wait()
            torch.cuda.synchronize()
            print(f"rank {self.rank} isend non-default group done")

        else:
            op = dist.P2POp(dist.irecv, tensor_recv_shape, self.rank-2)
            reqs = dist.batch_isend_irecv([op])
            for req in reqs:
                req.wait()
            torch.cuda.synchronize()
            print(f"rank {self.rank} irecv default group done")
            
            print(f"rank {self.rank} irecv non-default group begin")
            op = dist.P2POp(dist.irecv, tensor_recv, self.rank-2, self.grp)
            reqs = dist.batch_isend_irecv([op])
            for req in reqs:
                req.wait()
            torch.cuda.synchronize()
            print(f"rank {self.rank} irecv non-default group done")


if __name__ == "__main__":
    bug = Bug()
    bug.run()
```

The launch script.
```
# Launch script
#!/bin/bash

export CUDA_DEVICE_MAX_CONNECTIONS=1
export NCCL_SOCKET_IFNAME=<Your NIC>
export GLOO_SOCKET_IFNAME=<Your NIC>

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=COLL

export GPUS_PER_NODE=${GPUS_PER_NODE:-2}
export MASTER_ADDR=${MASTER_ADDR:-<Your master address>}
export MASTER_PORT=6000
export NNODES=${NNODES:-3}
export NODE_RANK=${NODE_RANK:-0}
export WORLD_SIZE=$(($GPUS_PER_NODE*$NNODES))

DISTRIBUTED_ARGS="
    --nproc_per_node $GPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT
"

torchrun $DISTRIBUTED_ARGS test.py
```

## Trace Details

- **Trace Observation**: The trace log indicates that the `ncclGroupEnd` call takes an unusually long time (until the kill of the process) after the `ncclCommInitRankConfig` call for the second group, as shown in the trace screenshot provided.

<img width="931" alt="Screenshot 2024-01-01 at 13 34 13" src="https://github.com/pytorch/pytorch/assets/28251862/0ec3c9fa-ddc3-4e42-b071-62729e4be1f6">


### Versions

PyTorch version: 2.1.0a0+32f93b1
Is debug build: False
CUDA used to build PyTorch: 12.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.27.4
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-167-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.128
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe

Nvidia driver version: 535.86.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7313 16-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 2
Core(s) per socket:                 16
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        3000.0000
CPU min MHz:                        1500.0000
BogoMIPS:                           5999.78
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca
Virtualization:                     AMD-V
L1d cache:                          1 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           16 MiB (32 instances)
L3 cache:                           256 MiB (8 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-15,32-47
NUMA node1 CPU(s):                  16-31,48-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] onnx==1.14.0
[pip3] pytorch-quantization==2.1.2
[pip3] torch==2.1.0a0+32f93b1
[pip3] torch-tensorrt==2.0.0.dev0
[pip3] torchdata==0.7.0a0
[pip3] torchtext==0.16.0a0
[pip3] torchvision==0.16.0a0
[pip3] triton==2.1.0+e621604
[conda] Could not collect

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225