Repository: pytorch
Issue: Abort called in FSDP tests
Labels: oncall: distributed,module: fsdp

### 🐛 Describe the bug

When running the test suite of PyTorch 1.12.1 I get (e.g.)

```
distributed/fsdp/test_fsdp_input failed!
distributed/fsdp/test_fsdp_mixed_precision failed!
```

Tracing this down to the origin I see:
```
terminate called without an active exception
SIGABRT(6), PID: 2450054, Thread 2450054: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
<omitting python frames>
frame #14: __libc_start_main + 0xf3 (0x14f861828cf3 in /lib64/libc.so.6)
frame #15: _start + 0x2e (0x40106e in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/Python/3.10.4-GCCcore-11.3.0/bin/python)

SIGABRT(6), PID: 2450054, Thread 2450055: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #2: __poll + 0x51 (0x14f861912ac1 in /lib64/libc.so.6)
frame #3: <unknown function> + 0x292929 (0x14f831ff4929 in /lib64/libcuda.so.1)
frame #4: <unknown function> + 0x3481f4 (0x14f8320aa1f4 in /lib64/libcuda.so.1)
frame #5: <unknown function> + 0x28e3a8 (0x14f831ff03a8 in /lib64/libcuda.so.1)
frame #6: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #7: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)

SIGABRT(6), PID: 2450054, Thread 2450056: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #2: pthread_cond_timedwait + 0x25a (0x14f861dc579a in /lib64/libpthread.so.0)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdogInternal() + 0x4ec (0x14f82098e01c in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x79 (0x14f82098eef9 in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdfa44 (0x14f8337c1a44 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libstdc++.so.6)
frame #6: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #7: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)

SIGABRT(6), PID: 2450054, Thread 2450057: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #2: __poll + 0x51 (0x14f861912ac1 in /lib64/libc.so.6)
frame #3: <unknown function> + 0x292929 (0x14f831ff4929 in /lib64/libcuda.so.1)
frame #4: <unknown function> + 0x3481f4 (0x14f8320aa1f4 in /lib64/libcuda.so.1)
frame #5: <unknown function> + 0x28e3a8 (0x14f831ff03a8 in /lib64/libcuda.so.1)
frame #6: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #7: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)

SIGABRT(6), PID: 2450054, Thread 2450058: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #2: __read + 0x44 (0x14f861dc8aa4 in /lib64/libpthread.so.0)
frame #3: ibv_get_async_event + 0x33 (0x14f81dbf5f73 in /lib64/libibverbs.so.1)
frame #4: <unknown function> + 0x640f2 (0x14f80d1530f2 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0/lib/libnccl.so.2)
frame #5: <unknown function> + 0x74c4b (0x14f80d163c4b in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0/lib/libnccl.so.2)
frame #6: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #7: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)

SIGABRT(6), PID: 2450054, Thread 2450061: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #2: __poll + 0x51 (0x14f861912ac1 in /lib64/libc.so.6)
frame #3: <unknown function> + 0x5ccd2 (0x14f80d14bcd2 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0/lib/libnccl.so.2)
frame #4: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #5: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)

SIGABRT(6), PID: 2450054, Thread 2450062: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #2: pthread_cond_wait + 0x1fc (0x14f861dc544c in /lib64/libpthread.so.0)
frame #3: <unknown function> + 0x5c410 (0x14f80d14b410 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0/lib/libnccl.so.2)
frame #4: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #5: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)

SIGABRT(6), PID: 2450054, Thread 2450064: 
 frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x5f (0x14f81facb4df in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x15a (0x14f81facb8fa in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x12ce0 (0x14f861dc9ce0 in /lib64/libpthread.so.0)
frame #3: gsignal + 0x10f (0x14f86183ca9f in /lib64/libc.so.6)
frame #4: abort + 0x127 (0x14f86180fe05 in /lib64/libc.so.6)
frame #5: <unknown function> + 0xa996a (0x14f83378b96a in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libstdc++.so.6)
frame #6: <unknown function> + 0xb51ca (0x14f8337971ca in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libstdc++.so.6)
frame #7: <unknown function> + 0xb5235 (0x14f833797235 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libstdc++.so.6)
frame #8: __gxx_personality_v0 + 0x2bc (0x14f833796b8c in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libstdc++.so.6)
frame #9: <unknown function> + 0x115c4 (0x14f8487595c4 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libgcc_s.so.1)
frame #10: _Unwind_ForcedUnwind + 0x132 (0x14f848759cb2 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libgcc_s.so.1)
frame #11: __pthread_unwind + 0x46 (0x14f861dc8616 in /lib64/libpthread.so.0)
frame #12: <unknown function> + 0x945b (0x14f861dc045b in /lib64/libpthread.so.0)
<omitting python frames>
frame #16: <unknown function> + 0x33e265 (0x14f83130b265 in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #17: <unknown function> + 0x602666 (0x14f8315cf666 in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #18: <unknown function> + 0x339f1a (0x14f831306f1a in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #19: <unknown function> + 0x602362 (0x14f8315cf362 in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0x34a28e6 (0x14f82ace38e6 in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #21: <unknown function> + 0xf602fa (0x14f8287a12fa in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #22: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x16d (0x14f82acde5ad in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #23: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x80 (0x14f82acd6360 in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #24: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x5e (0x14f8315d0cae in /tmp/eb-6mbkb9je/tmpaivgfiz_/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #25: <unknown function> + 0xdfa44 (0x14f8337c1a44 in /user/gent/400/vsc40023/eb_scratch/RHEL8/zen3-ampere-ib/software/GCCcore/11.3.0/lib64/libstdc++.so.6)
frame #26: <unknown function> + 0x81cf (0x14f861dbf1cf in /lib64/libpthread.so.0)
frame #27: clone + 0x43 (0x14f861827dd3 in /lib64/libc.so.6)
```

There are some missing symbols but the main point is "terminate called without an active exception".

I've seen similar `abort` calls in e.g. TensorFlow when `pybind11::gil_scoped_release` is used in destructors as it is not allowed to call the GIL release function when Python is finalizing. With NCCL in the traceback I found [`destroy_nccl_comm`](https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/python_nccl.cpp#L46) as a potential candidate for such an issue: The `nccl_init_rank` creates a capsule with that function set as the "destructor" and it might be possible that it is destroyed when Python gets finalized.

So I'd suggest to visit such uses and protect them, see https://github.com/pybind/pybind11/commit/79cb013f1f67d5565700e528cb5d8ef079fe5e8a and the PR https://github.com/pybind/pybind11/pull/2657 which explicitly discusses PyTorch.

### Versions

1.12.1

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu