Repository: pytorch
Issue: segfault during shutdown with torch1.7
Labels: module: dependency bug,module: crash,triaged,module: pybind,module: third_party

## 🐛 Bug

I'm seeing a sporadic crash in one of my unit tests due to PyThreadState_GET() returning NULL while the GIL is held


## To Reproduce

Steps to reproduce the behavior:
1. Install torch 1.7.0 (I used python 3.7.8) and pytest
2. Checkout https://github.com/froody/fairscale/tree/torch17_crash
3. execute `while true; do pytest -k "test_python_autograd_function[PipelineStyle.MultiProcess]" -s || break; done`
4. wait for it to crash

Example backtrace:

```
#0  subtype_dealloc () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/typeobject.c:1191
#1  0x000055adf4a75305 in insertdict (value=None, hash=<optimized out>, key=<optimized out>, mp=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/dictobject.c:1076
#2  PyDict_SetItem (value=None, key=<optimized out>, op={'__name__': None, '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None, 'gethostbyname': <built-in method gethostbyname of module object at remote 0x7f8b4e39f590>, 'gethostbyname_ex': <built-in method gethostbyname_ex of module object at remote 0x7f8b4e39f590>, 'gethostbyaddr': <built-in method gethostbyaddr of module object at remote 0x7f8b4e39f590>, 'gethostname': <built-in method gethostname of module object at remote 0x7f8b4e39f590>, 'sethostname': <built-in method sethostname of module object at remote 0x7f8b4e39f590>, 'getservbyname': <built-in method getservbyname of module object at remote 0x7f8b4e39f590>, 'getservbyport': <built-in method getservbyport of module object at remote 0x7f8b4e39f590>, 'getprotobyname': <built-in method getprotobyname of module object at remote 0x7f8b4e39f590>, 'close': <built-in method close of module object at remote 0x7f8b4e39f590>, 'dup': <built-in method dup of module object at remote 0x7f8b4e39f590>, 'socketpair': <built-in method socketpair of ...(truncated)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/dictobject.c:1463
#3  _PyModule_ClearDict () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/moduleobject.c:626
#4  0x000055adf4adcc0c in PyImport_Cleanup () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/import.c:558
#5  0x000055adf4b50d04 in Py_FinalizeEx () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pylifecycle.c:1195
#6  0x000055adf4b50e38 in Py_Exit (sts=0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pylifecycle.c:2288
#7  0x000055adf4b50eee in handle_system_exit () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:636
#8  0x000055adf4b50f7c in PyErr_PrintEx () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:646
#9  0x000055adf4b62202 in PyRun_SimpleStringFlags () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:457
#10 0x000055adf4b622d9 in pymain_run_command (cf=0x7ffc556d0fb0, command=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:414
#11 pymain_run_python (pymain=0x7ffc556d10c0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:2901
#12 pymain_main () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:3068
#13 0x000055adf4b6267c in _Py_UnixMain () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:3103
#14 0x00007f8b4f00bb97 in __libc_start_main (main=0x55adf4a21240 <main>, argc=4, argv=0x7ffc556d1218, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc556d1208) at ../csu/libc-start.c:310
#15 0x000055adf4b07101 in _start ()
```


## Expected behavior

test passes

## Proposed fix

Replace [this code](https://github.com/pybind/pybind11/blob/4038542b157d8e7ffd5c6f47166d9544fea03800/include/pybind11/pybind11.h#L2118-L2120) with :
```
if(!_Py_IsFinalizing()) {
    PyThreadState_Clear(tstate);
    PyThreadState_DeleteCurrent();
    PYBIND11_TLS_DELETE_VALUE(detail::get_internals().tstate);
}
```

## Environment

PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.105
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
GPU 4: Tesla V100-SXM2-32GB
GPU 5: Tesla V100-SXM2-32GB
GPU 6: Tesla V100-SXM2-32GB
GPU 7: Tesla V100-SXM2-32GB

Nvidia driver version: 418.116.00
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.0
[pip3] torch-pg==0.0.0
[pip3] torchaudio==0.7.0a0+ac17b64
[pip3] torchvision==0.8.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h6bb024c_0
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py37h8f50634_2    conda-forge
[conda] mkl_fft                   1.2.0            py37h161383b_1    conda-forge
[conda] mkl_random                1.2.0            py37h9fdb41a_1    conda-forge
[conda] numpy                     1.19.2           py37h54aff64_0
[conda] numpy-base                1.19.2           py37hfa32c7d_0
[conda] pytorch                   1.7.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torch-pg                  0.0.0                    pypi_0    pypi
[conda] torchaudio                0.7.0                      py37    pytorch
[conda] torchvision               0.8.1                py37_cu101    pytorch

## Additional context
I tried debugging this with GDB and I think I found the root cause, consider [this gdb session](https://gist.github.com/froody/d90c683b32f8609e01fad55216781cb9)

1. (line 5) thread 1 suspends normally, setting PyThreadState = NULL
2. (line 48) thread 1 drops the GIL 
3. (line 87) thread 18 acquires the GIL
4. (line 124) thread 18 drops the GIL again due to _Py_IsFinalizing
5. (line 162) thread 1 acquires the GIL as part of resume
6. thread 18 calls PyThreadState_DeleteCurrent() without holding the GIL, while PyThreadState is NULL, aborting

I believe this is due to the `pybind11::gil_scoped_acquire` created by thread18 being destructed as part of stack unwinding, and not checking the value of _Py_IsFinalizing

I believe this could cause the crash mentioned at the top because thread 18 could call  PyThreadState_DeleteCurrent() without the GIL while thread 1 executes cleanup functions while holding the GIL as part of Py_FinalizeEx. In this crash I assume the order of events is something like

1. Thread 18 constructs a pybind11::gil_scoped_acquire and acquires the GIL
2. Thread 18 drops the GIL as part of PyEval_EvalFrameEx to allow preemption
3. Thread 1 acquires the GIL and resumes
4. Thread 1 calls Py_FinalizeEx
5. Thread 1 drops the GIL as part of PyEval_EvalFrameEx
6. Thread 18 attempts to resume but _Py_IsFinalizing() returns True so it calls pthread_exit()
7. Thread 1 acquires the GIL and resumes due to _Py_CURRENTLY_FINALIZING
8. Thread 18 unwinds the stack without holding the GIL, and `~pybind11::gil_scoped_acquire` calls PyThreadState_DeleteCurrent() without holding the GIL
9. Thread 1 calls PyThreadState_GET() in subtype_dealloc while holding the GIL and gets NULL, de-references it and crashes



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski