Repository: pytorch
Issue: Sporadic CUDA error in `test_nccl_warn_not_in_group_debug_detail`
Labels: oncall: distributed

Running
```
CUDA_LAUNCH_BLOCKING=1 CUDA_VISIBLE_DEVICES=0,7 numactl -C 2 python test/distributed/test_c10d_nccl.py -k test_nccl_warn_not_in_group_debug_detail --repeat 20
```
fails sporadically with error like:
```
Exception raised from c10_cuda_check_implementation at /fsx/users/andgu/work/pytorch/c10/cuda/CUDAException.cpp:44 (most recent call first):
```

<details>
<summary> Full stack trace </summary>

```
INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3154296
INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3154297
INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
[E ProcessGroupNCCL.cpp:830] [Rank 0] NCCL watchdog thread terminated with exception: CUDA error: driver shutting down
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /fsx/users/andgu/work/pytorch/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f5a5cea89ec in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xfa (0x7f5a5ce6c65a in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3cc (0x7f5a5cf3d24c in /fsx/users/andgu/work/pytorch/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::startedGPUExecutionInternal() const + 0x6c (0x7f5a5ddaf86c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isStarted() + 0x78 (0x7f5a5ddb3388 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x201 (0x7f5a5ddc15e1 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x8c (0x7f5a5ddc187c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 0] NCCL watchdog thread terminated with exception: CUDA error: driver shutting down
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /fsx/users/andgu/work/pytorch/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f5a5cea89ec in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xfa (0x7f5a5ce6c65a in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3cc (0x7f5a5cf3d24c in /fsx/users/andgu/work/pytorch/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::startedGPUExecutionInternal() const + 0x6c (0x7f5a5ddaf86c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isStarted() + 0x78 (0x7f5a5ddb3388 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x201 (0x7f5a5ddc15e1 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x8c (0x7f5a5ddc187c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154296: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: std::vector<c10::Type::SingletonOrSharedTypePtr<c10::Type>, std::allocator<c10::Type::SingletonOrSharedTypePtr<c10::Type> > >::~vector() + 0 (0x7f5a729b6140 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_python.so)
frame #3: <unknown function> + 0x15d122c (0x7f5a67a3a22c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #4: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x48 (0x7f5a72651178 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x15d1ed4 (0x7f5a67a3aed4 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #6: std::_Sp_counted_ptr_inplace<torch::jit::Operator, std::allocator<torch::jit::Operator>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x50d (0x7f5a6b20b76d in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d9bca8 (0x7f5a6b204ca8 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x468a7 (0x7f5aa414c8a7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: on_exit + 0 (0x7f5aa414ca60 in /lib/x86_64-linux-gnu/libc.so.6)
frame #10: <unknown function> + 0x1168d7 (0x561c0ca438d7 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #11: <unknown function> + 0x116903 (0x561c0ca43903 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #12: <unknown function> + 0x116952 (0x561c0ca43952 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #13: PyRun_SimpleStringFlags + 0x4d (0x561c0ca44dae in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #14: <unknown function> + 0x118fdf (0x561c0ca45fdf in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #15: Py_BytesMain + 0x39 (0x561c0cb84729 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #16: __libc_start_main + 0xf3 (0x7f5aa412a083 in /lib/x86_64-linux-gnu/libc.so.6)
frame #17: <unknown function> + 0x1e6995 (0x561c0cb13995 in /fsx/users/andgu/conda/envs/pytorch/bin/python)

SIGABRT(6), PID: 3154296, Thread 3154350: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f5aa421899f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x292ec9 (0x7f5a73582ec9 in /lib/x86_64-linux-gnu/libcuda.so)
frame #4: <unknown function> + 0x34d9ab (0x7f5a7363d9ab in /lib/x86_64-linux-gnu/libcuda.so)
frame #5: <unknown function> + 0x2957f8 (0x7f5a735857f8 in /lib/x86_64-linux-gnu/libcuda.so)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154386: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x152 (0x7f5a5ceafa62 in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #2: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: gsignal + 0xcb (0x7f5aa414900b in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: abort + 0x12b (0x7f5aa4128859 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: __gnu_cxx::__verbose_terminate_handler() + 0xbc (0x7f5a750ed84a in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0xabf47 (0x7f5a750ebf47 in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #7: <unknown function> + 0xabf7d (0x7f5a750ebf7d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0xabf44 (0x7f5a750ebf44 in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #9: <unknown function> + 0xc5e08a (0x7f5a5dbbf08a in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #11: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #12: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154387: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: epoll_wait + 0x5e (0x7f5aa422546e in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: gloo::transport::tcp::Loop::run() + 0x6a (0x7f5a6d8def2a in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154391: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f5aa4307376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: std::condition_variable::wait(std::unique_lock<std::mutex>&) + 0x9 (0x7f5a751044cb in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #4: c10d::ProcessGroupGloo::runLoop(int) + 0x221 (0x7f5a6b7e0371 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154392: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f5aa4307376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: std::condition_variable::wait(std::unique_lock<std::mutex>&) + 0x9 (0x7f5a751044cb in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #4: c10d::ProcessGroupGloo::runLoop(int) + 0x221 (0x7f5a6b7e0371 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154395: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_timedwait + 0x271 (0x7f5aa43077d1 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0xef (0x7f5a5ddc14cf in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x8c (0x7f5a5ddc187c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

[E ProcessGroupNCCL.cpp:830] [Rank 1] NCCL watchdog thread terminated with exception: CUDA error: driver shutting down
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /fsx/users/andgu/work/pytorch/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f12c6fb69ec in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xfa (0x7f12c6f7a65a in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3cc (0x7f12c704b24c in /fsx/users/andgu/work/pytorch/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::startedGPUExecutionInternal() const + 0x6c (0x7f12c7ebd86c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isStarted() + 0x78 (0x7f12c7ec1388 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x201 (0x7f12c7ecf5e1 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x8c (0x7f12c7ecf87c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xc819d (0x7f12df21619d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] NCCL watchdog thread terminated with exception: CUDA error: driver shutting down
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /fsx/users/andgu/work/pytorch/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f12c6fb69ec in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xfa (0x7f12c6f7a65a in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3cc (0x7f12c704b24c in /fsx/users/andgu/work/pytorch/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::startedGPUExecutionInternal() const + 0x6c (0x7f12c7ebd86c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isStarted() + 0x78 (0x7f12c7ec1388 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x201 (0x7f12c7ecf5e1 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x8c (0x7f12c7ecf87c in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xc819d (0x7f12df21619d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154396: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: epoll_wait + 0x5e (0x7f5aa422546e in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: gloo::transport::tcp::Loop::run() + 0x6a (0x7f5a6d8def2a in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154397: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f5aa4307376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: std::condition_variable::wait(std::unique_lock<std::mutex>&) + 0x9 (0x7f5a751044cb in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #4: c10d::ProcessGroupGloo::runLoop(int) + 0x221 (0x7f5a6b7e0371 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154398: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f5aa4307376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: std::condition_variable::wait(std::unique_lock<std::mutex>&) + 0x9 (0x7f5a751044cb in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #4: c10d::ProcessGroupGloo::runLoop(int) + 0x221 (0x7f5a6b7e0371 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0xc819d (0x7f5a7510819d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154399: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f5aa421899f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x292ec9 (0x7f5a73582ec9 in /lib/x86_64-linux-gnu/libcuda.so)
frame #4: <unknown function> + 0x34d9ab (0x7f5a7363d9ab in /lib/x86_64-linux-gnu/libcuda.so)
frame #5: <unknown function> + 0x2957f8 (0x7f5a735857f8 in /lib/x86_64-linux-gnu/libcuda.so)
frame #6: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154502: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f5aa421899f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x308ab43 (0x7f5a5ffebb43 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154503: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f5aa4307376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: <unknown function> + 0x308a150 (0x7f5a5ffeb150 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154524: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f5aa421899f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x308ab43 (0x7f5a5ffebb43 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154296, Thread 3154560: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f5a5ceaf51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f5aa430c420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f5aa4307376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: <unknown function> + 0x308a150 (0x7f5a5ffeb150 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x8609 (0x7f5aa4300609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f5aa4225133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154297: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: c10::impl::OperatorEntry::getKernelForDispatchKey(c10::DispatchKey) const + 0x38 (0x7f12d1ae9088 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #3: c10::impl::OperatorEntry::computeDispatchTableEntryWithDebug(c10::Dispatcher const&, c10::DispatchKey) const + 0xb8 (0x7f12d1aea338 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #4: c10::impl::OperatorEntry::computeDispatchTableEntry(c10::Dispatcher const&, c10::DispatchKey) const + 0xd (0x7f12d1aea55d in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: c10::impl::OperatorEntry::updateDispatchTableEntry_(c10::Dispatcher const&, c10::DispatchKey) + 0xa9 (0x7f12d1aea619 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #6: c10::impl::OperatorEntry::updateDispatchTable_(c10::Dispatcher const&, c10::DispatchKey) + 0x311 (0x7f12d1aeaa21 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #7: c10::impl::OperatorEntry::deregisterKernel_(c10::Dispatcher const&, c10::optional<c10::DispatchKey>, std::_List_iterator<c10::impl::AnnotatedKernel>) + 0x405 (0x7f12d1af04f5 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #8: c10::Dispatcher::deregisterImpl_(c10::OperatorHandle const&, c10::OperatorName const&, c10::optional<c10::DispatchKey>, std::_List_iterator<c10::impl::AnnotatedKernel>) + 0x52 (0x7f12d1adddd2 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0xe38abd (0x7f12c7ea7abd in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0x468a7 (0x7f130e25a8a7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #11: on_exit + 0 (0x7f130e25aa60 in /lib/x86_64-linux-gnu/libc.so.6)
frame #12: <unknown function> + 0x1168d7 (0x55bd6ef718d7 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #13: <unknown function> + 0x116903 (0x55bd6ef71903 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #14: <unknown function> + 0x116952 (0x55bd6ef71952 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #15: PyRun_SimpleStringFlags + 0x4d (0x55bd6ef72dae in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #16: <unknown function> + 0x118fdf (0x55bd6ef73fdf in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #17: Py_BytesMain + 0x39 (0x55bd6f0b2729 in /fsx/users/andgu/conda/envs/pytorch/bin/python)
frame #18: __libc_start_main + 0xf3 (0x7f130e238083 in /lib/x86_64-linux-gnu/libc.so.6)
frame #19: <unknown function> + 0x1e6995 (0x55bd6f041995 in /fsx/users/andgu/conda/envs/pytorch/bin/python)

SIGABRT(6), PID: 3154297, Thread 3154351: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f130e32699f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x292ec9 (0x7f12dd690ec9 in /lib/x86_64-linux-gnu/libcuda.so)
frame #4: <unknown function> + 0x34d9ab (0x7f12dd74b9ab in /lib/x86_64-linux-gnu/libcuda.so)
frame #5: <unknown function> + 0x2957f8 (0x7f12dd6937f8 in /lib/x86_64-linux-gnu/libcuda.so)
frame #6: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154389: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x152 (0x7f12c6fbda62 in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #2: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: gsignal + 0xcb (0x7f130e25700b in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: abort + 0x12b (0x7f130e236859 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: __gnu_cxx::__verbose_terminate_handler() + 0xbc (0x7f12df1fb84a in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0xabf47 (0x7f12df1f9f47 in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #7: <unknown function> + 0xabf7d (0x7f12df1f9f7d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0xabf44 (0x7f12df1f9f44 in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #9: <unknown function> + 0xc5e08a (0x7f12c7ccd08a in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0xc819d (0x7f12df21619d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #11: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #12: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154390: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: epoll_wait + 0x5e (0x7f130e33346e in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: gloo::transport::tcp::Loop::run() + 0x6a (0x7f12d79ecf2a in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0xc819d (0x7f12df21619d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154393: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f130e415376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: std::condition_variable::wait(std::unique_lock<std::mutex>&) + 0x9 (0x7f12df2124cb in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #4: c10d::ProcessGroupGloo::runLoop(int) + 0x221 (0x7f12d58ee371 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0xc819d (0x7f12df21619d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154394: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f130e415376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: std::condition_variable::wait(std::unique_lock<std::mutex>&) + 0x9 (0x7f12df2124cb in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #4: c10d::ProcessGroupGloo::runLoop(int) + 0x221 (0x7f12d58ee371 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0xc819d (0x7f12df21619d in /fsx/users/andgu/conda/envs/pytorch/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154400: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f130e32699f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x292ec9 (0x7f12dd690ec9 in /lib/x86_64-linux-gnu/libcuda.so)
frame #4: <unknown function> + 0x34d9ab (0x7f12dd74b9ab in /lib/x86_64-linux-gnu/libcuda.so)
frame #5: <unknown function> + 0x2957f8 (0x7f12dd6937f8 in /lib/x86_64-linux-gnu/libcuda.so)
frame #6: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154434: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f130e32699f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x292ec9 (0x7f12dd690ec9 in /lib/x86_64-linux-gnu/libcuda.so)
frame #4: <unknown function> + 0x34d9ab (0x7f12dd74b9ab in /lib/x86_64-linux-gnu/libcuda.so)
frame #5: <unknown function> + 0x2957f8 (0x7f12dd6937f8 in /lib/x86_64-linux-gnu/libcuda.so)
frame #6: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154525: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: __poll + 0x4f (0x7f130e32699f in /lib/x86_64-linux-gnu/libc.so.6)
frame #3: <unknown function> + 0x308ab43 (0x7f12ca0f9b43 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

SIGABRT(6), PID: 3154297, Thread 3154559: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f12c6fbd51b in /fsx/users/andgu/work/pytorch/torch/lib/libc10.so)
frame #1: <unknown function> + 0x14420 (0x7f130e41a420 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #2: pthread_cond_wait + 0x216 (0x7f130e415376 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #3: <unknown function> + 0x308a150 (0x7f12ca0f9150 in /fsx/users/andgu/work/pytorch/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x8609 (0x7f130e40e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f130e333133 in /lib/x86_64-linux-gnu/libc.so.6)

F
======================================================================
FAIL: test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/fsx/users/andgu/work/pytorch/torch/testing/_internal/common_distributed.py", line 541, in wrapper
    self._join_processes(fn)
  File "/fsx/users/andgu/work/pytorch/torch/testing/_internal/common_distributed.py", line 760, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/fsx/users/andgu/work/pytorch/torch/testing/_internal/common_distributed.py", line 835, in _check_return_codes
    self.assertEqual(
  File "/fsx/users/andgu/work/pytorch/torch/testing/_internal/common_utils.py", line 3031, in assertEqual
    raise error_metas[0].to_error(
AssertionError: Scalars are not equal!

Expected 0 but got -6.
Absolute difference: 6
Relative difference: inf
Expected zero exit code but got -6 for pid: 3154296

----------------------------------------------------------------------
Ran 1 test in 16.997s

FAILED (failures=1)
```

</details>
Note that the `numactl -C 2` helps increase the failure rate.

Possibly, this is a duplicate of https://github.com/pytorch/pytorch/issues/90848 but needs an expert to confirm.


Thanks @Aidyn-A for flagging this!

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501