Repository: pytorch
Issue: ncclInternalError: Internal check failed
Labels: oncall: distributed,triaged,module: nccl

### 🐛 Describe the bug

I met an error when I use torchrun for 4 GPUs training and 'nccl' backend (It runs perfect when I use 'gloo'). The environment is python3.9+pytorch2.3.0+cuda12.1.We tried to use uftrace to capture the DLRM code of 4 GPUs launched by torchrun, the command is as follows:

**torchrun --nproc_per_node=4 ./multi-uftrace.py**
The multi-uftrace.py file content is as follows:

```
import subprocess
try:  
    result = subprocess.run([
    '/mnt/yuanningbai/local/uftrace/bin/uftrace','-e','record',
    '/mnt/yuanningbai/dlrm/dlrm_s_pytorch.py', '--mini-batch-size=4','--test-mini-batch-size=16384','--test-num-workers=0',
    '--num-batches=1','--data-generation=random','--arch-mlp-bot=512-512-64','--arch-sparse-feature-size=64','--arch-embedding-size=1000000-1000000-1000000-1000000-1000000-1000000-1000000-1000000','--num-indices-per-lookup=100',
    '--arch-interaction-op=dot','--print-freq=1','--print-time','--use-gpu','--inference-only','--dist-backend=nccl'], 
    check=True,capture_output=True, text=True)#  
except subprocess.CalledProcessError as e:  
    print("error code :", e.returncode)  
    print("error info :", e.output)  

```


## The error message is as follows:


```
W1029 16:52:13.175227 140626680026112 torch/distributed/run.py:757]
W1029 16:52:13.175227 140626680026112 torch/distributed/run.py:757] *****************************************
W1029 16:52:13.175227 140626680026112 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1029 16:52:13.175227 140626680026112 torch/distributed/run.py:757] *****************************************
error code ： 1
error info ： world size: 4, current rank: 1, local rank: 1
error code ： 1
error info ： world size: 4, current rank: 3, local rank: 3
error code ： 1
error info ： Running on 4 ranks using nccl backend
fail to enable all_to_all_single primitive: NCCL error in: /mnt/yuanningbai/pytorch-2.3.0/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, internal error - please report this issue to the NCCL developers, NCCL version 2.20.5
ncclInternalError: Internal check failed.
Last error:
Error : ring 8 does not loop back to start (1 != 0)
world size: 4, current rank: 0, local rank: 0
Using 1 GPU(s)...
-*-*-*-*-*-*nn.EmbeddingBag-*-*-*-*-*-*
-*-*-*-*-*-*nn.EmbeddingBag-*-*-*-*-*-*
error code ： 1
error info ： world size: 4, current rank: 2, local rank: 2
```

In order to capture the underlying functions of pytorch, we compile pytorch into the pg version. The above error will occur under 4 GPUs, but not under 2 GPUs. At the same time, we try to compile it into the develop version and it will run correctly. So I would like to ask if there is any solution to prevent such errors under the 4 GPUs of the pg version?

### Versions

GPU：  4 x A100 80G GPU
Driver Version ：530.30.02
CUDA Version ： 12.1
OS version ：Ubuntu 22.04
python :3.9
pytorch :v2.3.0
nccl: v2.20.5

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o