Repository: pytorch
Issue: Unused `import torch` followed by `cuml.NearestNeighbors` leads to nondeterministic segfault (during Python process exit?)
Labels: needs reproduction,module: crash,triaged

### 🐛 Describe the bug

I've also filed this at https://github.com/rapidsai/cuml/issues/5343, since it appears there is some interaction between an (unused) `import torch` and a `cuml.NearestNeighbors` call. There's also more details (including instructions and templates for launching an EC2 GPU node that repros the issue) at [ryan-williams/torch-cuml-gpu-segfault](https://github.com/ryan-williams/torch-cuml-gpu-segfault).

Below is a summary of the repro; I've run into it on AWS `p3.2xlarge` instances, with a variety of "Deep Learning" AMIs published by Amazon and NVIDIA, and using Pytorch 1.12.1 and 1.13.1.

`neighbors.py`:
```python
import numpy as np
np.random.seed(123)
X = np.random.random((10, 2))

# This unused import, when executed before the cuml block below, leads to a segfault
# (seemingly during Python process cleanup) on ≈10% of runs.
import torch

# Adapted from scanpy.neighbors.compute_neighbors_rapids:
# https://github.com/scverse/scanpy/blob/1.8.2/scanpy/neighbors/__init__.py#L318-L338
from cuml.neighbors import NearestNeighbors
nn = NearestNeighbors(n_neighbors=X.shape[1], metric='euclidean')
X_contiguous = np.ascontiguousarray(X, dtype=np.float32)
nn.fit(X_contiguous)
# This line also required for segfault
knn_indices, knn_distances = nn.kneighbors(X_contiguous)

# This print always occurs, segfault happens later (during Python process exit?)
print('Done!')
```
</details>

Exercising this code repeatedly exits 139 (segmentation fault) about ≈10% of the time:
```bash
n=30
for i in `seq $n`; do
    python neighbors.py && echo "$i/$n: ✅" || echo "$i/$n: ❌ ($?)"
done
# Done!
# 1/30: ✅
# Done!
# 2/30: ✅
# Done!
# 3/30: ✅
# Done!
# 4/30: ✅
# Done!
# 5/30: ✅
# Done!
# Segmentation fault (core dumped)
# 6/30: ❌ (139)
# Done!
# 7/30: ✅
# Done!
# 8/30: ✅
# Done!
# 9/30: ✅
# Done!
# 10/30: ✅
# Done!
# 11/30: ✅
# Done!
# 12/30: ✅
# Done!
# 13/30: ✅
# Done!
# 14/30: ✅
# Done!
# 15/30: ✅
# Done!
# 16/30: ✅
# Done!
# Segmentation fault (core dumped)
# 17/30: ❌ (139)
# Done!
# 18/30: ✅
# Done!
# 19/30: ✅
# Done!
# Segmentation fault (core dumped)
# 20/30: ❌ (139)
# Done!
# 21/30: ✅
# Done!
# 22/30: ✅
# Done!
# Segmentation fault (core dumped)
# 23/30: ❌ (139)
# Done!
# 24/30: ✅
# Done!
# Segmentation fault (core dumped)
# 25/30: ❌ (139)
# Done!
# 26/30: ✅
# Done!
# 27/30: ✅
# Done!
# 28/30: ✅
# Done!
# 29/30: ✅
# Done!
# 30/30: ✅
```

`environment.yml`:
```yml
channels:
  - pytorch
  - rapidsai
  - nvidia
  - conda-forge
dependencies:
  - python==3.9.13
  - cuml==22.12.00[build=cuda11_py39*]
  - numpy==1.23.5
  - pytorch==1.13.1
```

<details><summary><code>nvidia-smi</code></summary>

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   32C    P0    48W / 300W |   2323MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     43189      C   python                           2321MiB |
+-----------------------------------------------------------------------------+
```
</details> 

I originally hit this issue with CUDA 11.6, `cuml==22.06.01`, and Torch 1.12.1, but the repro persists on CUDA 11.7, `cuml==22.12.00`, and Torch 1.13.1, as documented here.

<details><summary>Here's <code>nvidia-smi</code> from a CUDA 11.6 / Driver Version 510.47.03 AMI where I first saw the error</summary>

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   26C    P0    24W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```
</details> 

### Versions

Here are two environments where I see the issue:

<details><summary><code>python collect_env.py</code> (Pytorch version 1.13.1 / cudatoolkit 11.7.0)</summary>

```
Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Clang version: Could not collect
CMake version: version 3.22.4
Libc version: glibc-2.26

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.14.256-197.484.amzn2.x86_64-x86_64-with-glibc2.26
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 450.142.00
cuDNN version: Probably one of the following:
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:            1
CPU MHz:             2697.660
CPU max MHz:         3000.0000
CPU min MHz:         1200.0000
BogoMIPS:            4600.03
Hypervisor vendor:   Xen
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            46080K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[conda] blas                      1.0                         mkl    conda-forge
[conda] cudatoolkit               11.7.0              hd8887f6_10    nvidia
[conda] mkl                       2023.0.0         h6d00ec8_25399
[conda] numpy                     1.23.5           py39h3d75532_0    conda-forge
[conda] pytorch                   1.13.1          py3.9_cuda11.7_cudnn8.5.0_0    pytorch
[conda] pytorch-cuda              11.7                 h67b0de4_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
```
</details> 

<details><summary><code>python collect_env.py</code> (Pytorch version 1.12.1 / cudatoolkit 11.6.0)</summary>

```
Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.26.3
Libc version: glibc-2.31

Python version: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 515.48.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           79
Model name:                      Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:                        1
CPU MHz:                         3000.000
CPU max MHz:                     3000.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        4600.01
Hypervisor vendor:               Xen
Virtualization type:             full
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        1 MiB
L3 cache:                        45 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] numpyro==0.11.0
[pip3] pytorch-lightning==1.7.7
[pip3] torch==1.12.1
[pip3] torch-cluster==1.6.0
[pip3] torch-geometric==2.1.0.post1
[pip3] torch-scatter==2.1.0
[pip3] torch-sparse==0.6.16
[pip3] torchmetrics==0.10.1
[conda] blas                      1.0                         mkl    conda-forge
[conda] cudatoolkit               11.6.0               habf752d_9    nvidia
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.1.0           hc2b9512_224
[conda] numpy                     1.22.4           py39hc58783e_0    conda-forge
[conda] numpyro                   0.11.0                   pypi_0    pypi
[conda] pyg                       2.1.0           py39_torch_1.12.0_cu116    pyg
[conda] pytorch                   1.12.1          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cluster           1.6.0           py39_torch_1.12.0_cu116    pyg
[conda] pytorch-lightning         1.7.7                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-scatter           2.1.0           py39_torch_1.12.0_cu116    pyg
[conda] pytorch-sparse            0.6.16          py39_torch_1.12.0_cu116    pyg
[conda] torchmetrics              0.10.1                   pypi_0    pypi
```
</details>