Repository: pytorch
Issue: CUDA error: an illegal memory access was encountered when use TORCH_NCCL_NAN_CHECK.
Labels: oncall: distributed,triaged,module: c10d

The script to reproduce the bug.

```python
import os
import time
import pickle
import torch
import threading
import torch.distributed as dist
import torch.distributed.distributed_c10d as c10d
from datetime import timedelta


def test_trace_dump():
    os.environ["TORCH_NCCL_NAN_CHECK"] = "1"

    dist.init_process_group("nccl", timeout=timedelta(seconds=60))
    rank = dist.get_rank()
    device = torch.device(rank)
    print(f"Run the example on rank {rank}")

    # Start the thread to monitor nccl trace
    dim = 1024
    a = torch.full((dim, dim), float(rank), device=device)
    pg = c10d._get_default_group()
    pg.allreduce(a).wait()

    torch.cuda.synchronize(device=device)
    dist.destroy_process_group()


if __name__ == "__main__":
    test_trace_dump()

```

 Run the example by torchrun with 2 GPUs like `torchrun --standalone --nproc-per-node=2 test_script.py`. Then, we can the full trace back like

```
n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1

n122-164-108:7400:7642 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7642 [1] NCCL INFO include/alloc.h:246 -> 1
n122-164-108:7400:7642 [1] NCCL INFO transport/net.cc:537 -> 1
n122-164-108:7400:7642 [1] NCCL INFO transport/net.cc:940 -> 1
n122-164-108:7400:7642 [1] NCCL INFO proxy.cc:980 -> 1
n122-164-108:7400:7642 [1] NCCL INFO proxy.cc:996 -> 1

n122-164-108:7400:7651 [1] include/alloc.h:125 NCCL WARN Cuda failure 700 'an illegal memory access was encountered'
n122-164-108:7400:7651 [1] NCCL INFO include/alloc.h:246 -> 1
n122-164-108:7400:7651 [1] NCCL INFO transport/p2p.cc:536 -> 1
n122-164-108:7400:7651 [1] NCCL INFO channel.cc:156 -> 1
n122-164-108:7400:7651 [1] NCCL INFO init.cc:209 -> 1
n122-164-108:7400:7651 [1] NCCL INFO init.cc:1933 -> 1

n122-164-108:7400:7651 [1] init.cc:2065 NCCL WARN commReclaim: cleanup comm 0x9e07a10 rank 1 failed in destroy/abort, error 1
n122-164-108:7400:7651 [1] NCCL INFO comm 0x9e07a10 rank 1 nranks 2 cudaDev 1 busId 6b020 - Abort COMPLETE
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f32d3473f86 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f32d3422d10 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f32d354ef08 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x1dc76 (0x7f32d3519c76 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x1f783 (0x7f32d351b783 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x1fac2 (0x7f32d351bac2 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
frame #6: <unknown function> + 0x5dbbc0 (0x7f3321001bc0 in /home/tiger/.local/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
```

### Versions

```
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.31

Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-5.10.135.bsk.6-amd64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H800
GPU 1: NVIDIA H800

Nvidia driver version: 535.129.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   52 bits physical, 57 bits virtual
CPU(s):                          180
On-line CPU(s) list:             0-179
Thread(s) per core:              2
Core(s) per socket:              45
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           143
Model name:                      Intel(R) Xeon(R) Platinum 8457C
Stepping:                        8
CPU MHz:                         2599.721
BogoMIPS:                        5199.44
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       4.2 MiB
L1i cache:                       2.8 MiB
L2 cache:                        180 MiB
L3 cache:                        195 MiB
NUMA node0 CPU(s):               0-89
NUMA node1 CPU(s):               90-179
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities

Versions of relevant libraries:
[pip3] byted-torch==2.1.0.post2
[pip3] byted-torch-monitor==0.0.1
[pip3] numpy==1.24.1
[pip3] onnx==1.16.1
[pip3] torch==2.4.0
[pip3] torchaudio==2.1.0+cu121
[pip3] torchlibrosa==0.1.0
[pip3] torchvision==0.16.0+cu121
[pip3] triton==3.0.0
```

cc @XilunWu @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o