Repository: pytorch
Issue: Stack smashing detected on nccl initialization
Labels: oncall: distributed,triaged,module: c10d

### 🐛 Describe the bug

I compiled latest main from source, during initialization of nccl i observe a stack smashing error.

I noticed that I compiled with USE_NUMPY OFF, after I turned it on I didn't see this issue again.

```
I0329 00:31:15.935866 140657074860032 object_utils.py:49] Removing "load_state_dict_only" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.935952 140657074860032 object_utils.py:49] Removing "distributed_training" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.936010 140657074860032 object_utils.py:49] Removing "profile_max_epochs" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.936059 140657074860032 object_utils.py:49] Removing "profile_check_val_every_n_epoch" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.936110 140657074860032 object_utils.py:49] Removing "profile_limit_train_batches" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.936152 140657074860032 object_utils.py:49] Removing "profile_limit_val_batches" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.936200 140657074860032 object_utils.py:49] Removing "timeout" from <class 'pytorch_lightning.trainer.trainer.Trainer'> constructor as it was not found in __init__.
I0329 00:31:15.936241 140657074860032 train.py:430] Using Pytorch Lightning trainer.
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.model_summary.ModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
[rank: 0] Seed set to 42
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
[W329 00:31:16.281428074 Utils.hpp:135] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarInt)
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------

*** stack smashing detected ***: terminated
[ip-10-198-119-35:1127336] *** Process received signal ***
[ip-10-198-119-35:1127336] Signal: Aborted (6)
[ip-10-198-119-35:1127336] Signal code:  (-6)
[ip-10-198-119-35:1127336] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fed46ffa520]
[ip-10-198-119-35:1127336] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fed4704e9fc]
[ip-10-198-119-35:1127336] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fed46ffa476]
[ip-10-198-119-35:1127336] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fed46fe07f3]
[ip-10-198-119-35:1127336] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x89676)[0x7fed47041676]
[ip-10-198-119-35:1127336] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x2a)[0x7fed470ee59a]
[ip-10-198-119-35:1127336] [ 6] /lib/x86_64-linux-gnu/libc.so.6(+0x136566)[0x7fed470ee566]
[ip-10-198-119-35:1127336] [ 7] /home/piotr/pytorch/torch/lib/libtorch_cuda.so(+0x6f50c9f)[0x7fec74365c9f]
[ip-10-198-119-35:1127336] [ 8] /home/piotr/pytorch/torch/lib/libtorch_cuda.so(+0x6f66d40)[0x7fec7437bd40]
[ip-10-198-119-35:1127336] [ 9] /home/piotr/pytorch/torch/lib/libtorch_cuda.so(+0x6f55b0c)[0x7fec7436ab0c]
[ip-10-198-119-35:1127336] [10] /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fed4704cac3]
[ip-10-198-119-35:1127336] [11] /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7fed470de850]
[ip-10-198-119-35:1127336] *** End of error message ***
fish: Job 1, 'train.py configs/bev/bev_model_…' terminated by signal SIGABRT (Abort)
```



BT:

```
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140650213131840) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140650213131840) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140650213131840, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fed46ffa476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fed46fe07f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fed47041676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fed4719392e "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007fed470ee59a in __GI___fortify_fail (msg=msg@entry=0x7fed47193916 "stack smashing detected") at ./debug/fortify_fail.c:26
#7  0x00007fed470ee566 in __stack_chk_fail () at ./debug/stack_chk_fail.c:24
#8  0x00007fec74365c9f in ncclInitKernelsForDevice (cudaArch=cudaArch@entry=860, maxStackSize=maxStackSize@entry=0x7febadfdbde8) at enqueue.cc:61
#9  0x00007fec7437bd40 in ncclCommInitRankFunc (job_=0x556f00000000) at init.cc:1370
#10 0x00007fec7436ab0c in ncclAsyncJobMain (arg=0x556f94fdec10) at group.cc:62
#11 0x00007fed4704cac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007fed470de850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
```

```
(gdb) p ncclDevKernelCount
$5 = 31
```

/home/piotr/pytorch/third_party/nccl/nccl/src/enqueue.cc
```
  46│       cudaFuncAttributes attr = {0};
  47│       CUDACHECKGOTO(cudaFuncGetAttributes(&attr, fn), result, ignore0);
  48│       if (attr.localSizeBytes > *maxStackSize) *maxStackSize = attr.localSizeBytes;
  49│     ignore0:;
  50│     }
  51│     if (carveout) {
  52│       CUDACHECKGOTO(cudaFuncSetAttribute(fn,
  53│         cudaFuncAttributePreferredSharedMemoryCarveout, carveout),
  54│         result, ignore1);
  55│     ignore1:;
  56│     }
  57│     if (ncclShmemDynamicSize(cudaArch) != 0) {
  58│       CUDACHECKGOTO(cudaFuncSetAttribute(fn,
  59│         cudaFuncAttributeMaxDynamicSharedMemorySize, ncclShmemDynamicSize(cudaArch)),
  60│         result, next_kernel);
  61├───> }
  62│   next_kernel:;
  63│   }
  64│   return result;
  65│ }
```

### Versions

Collecting environment information...
PyTorch version: 2.4.0a0+git3beb9d8
Is debug build: True
CUDA used to build PyTorch: 12.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1037-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 545.23.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          32
On-line CPU(s) list:             0-31
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7R32
CPU family:                      23
Model:                           49
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
Stepping:                        0
BogoMIPS:                        5599.99
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       512 KiB (16 instances)
L1i cache:                       512 KiB (16 instances)
L2 cache:                        8 MiB (16 instances)
L3 cache:                        64 MiB (4 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-31
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] flake8==7.0.0
[pip3] msgpack-numpy==0.4.8
[pip3] mypy==1.9.0
[pip3] mypy-boto3-ec2==1.34.73
[pip3] mypy-boto3-s3==1.28.55
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.4
[pip3] onnx==1.14.0
[pip3] onnx-graphsurgeon==0.3.27
[pip3] onnxsim==0.4.36
[pip3] optree==0.11.0
[pip3] pytorch-lightning==2.1.3
[pip3] pytorch-quantization==2.1.2
[pip3] torch==2.4.0a0+git3beb9d8
[pip3] torchmetrics==0.10.0
[pip3] torchpack==0.3.1
[pip3] torchvision==0.19.0a0+2c4665f
[conda] Could not collect

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k @rohan-varma