Repository: pytorch
Issue: CUSOLVER_STATUS_EXECUTION_FAILED when using the torch.logdet()
Labels: module: cuda,triaged,module: linear algebra

### 🐛 Describe the bug

My program reports an ERROR message when the ''torch.logdet()'' function was executed for the FIRST time. AMAZINGLY, the error will not be reported when I execute the function AGAIN. 

```python
Python 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> a = torch.randn(9, 9)
>>> a = a.cuda()
>>> a
tensor([[-2.6605, -1.5884,  0.7089,  0.7486, -0.5229,  1.6641,  1.2985, -2.0366,
          1.5453],
        [ 1.0522, -2.3295, -0.4025, -0.4814, -1.7643, -0.6771, -0.8490, -1.2713,
          0.2525],
        [ 0.2678, -0.9299, -1.2236, -0.6464,  1.1209, -0.3109,  1.4900,  1.3949,
          0.0460],
        [-0.5451, -0.6144, -0.3387, -0.2821, -0.7145,  0.6141, -0.5563,  0.5543,
          0.0465],
        [-0.9948, -0.8303,  1.1174,  1.5377,  0.8705, -0.5858, -0.1675, -0.2450,
         -0.0192],
        [ 0.7634, -0.8460, -0.4526, -0.5102, -0.2613, -0.3954,  0.1347, -0.7412,
         -0.2037],
        [ 0.9187,  1.4660,  1.2423,  0.8838, -0.0215, -0.4165,  0.4211, -1.8610,
         -0.7167],
        [ 0.4969, -0.8689, -0.4387,  1.0836,  1.1270, -0.2077,  0.1857, -1.9552,
          0.9171],
        [-0.5301, -0.8655, -0.2947, -1.4753,  1.1074,  1.4459, -0.1741,  0.5064,
         -2.0041]], device='cuda:0')
>>> a.requires_grad = True
>>> a
tensor([[-2.6605, -1.5884,  0.7089,  0.7486, -0.5229,  1.6641,  1.2985, -2.0366,
          1.5453],
        [ 1.0522, -2.3295, -0.4025, -0.4814, -1.7643, -0.6771, -0.8490, -1.2713,
          0.2525],
        [ 0.2678, -0.9299, -1.2236, -0.6464,  1.1209, -0.3109,  1.4900,  1.3949,
          0.0460],
        [-0.5451, -0.6144, -0.3387, -0.2821, -0.7145,  0.6141, -0.5563,  0.5543,
          0.0465],
        [-0.9948, -0.8303,  1.1174,  1.5377,  0.8705, -0.5858, -0.1675, -0.2450,
         -0.0192],
        [ 0.7634, -0.8460, -0.4526, -0.5102, -0.2613, -0.3954,  0.1347, -0.7412,
         -0.2037],
        [ 0.9187,  1.4660,  1.2423,  0.8838, -0.0215, -0.4165,  0.4211, -1.8610,
         -0.7167],
        [ 0.4969, -0.8689, -0.4387,  1.0836,  1.1270, -0.2077,  0.1857, -1.9552,
          0.9171],
        [-0.5301, -0.8655, -0.2947, -1.4753,  1.1074,  1.4459, -0.1741,  0.5064,
         -2.0041]], device='cuda:0', requires_grad=True)
>>> b= a+a
>>> b
tensor([[-5.3210, -3.1767,  1.4177,  1.4972, -1.0458,  3.3283,  2.5970, -4.0732,
          3.0906],
        [ 2.1044, -4.6591, -0.8051, -0.9628, -3.5285, -1.3542, -1.6980, -2.5426,
          0.5050],
        [ 0.5355, -1.8599, -2.4472, -1.2927,  2.2419, -0.6218,  2.9800,  2.7899,
          0.0920],
        [-1.0902, -1.2288, -0.6774, -0.5641, -1.4289,  1.2281, -1.1126,  1.1087,
          0.0929],
        [-1.9896, -1.6607,  2.2349,  3.0754,  1.7410, -1.1716, -0.3349, -0.4901,
         -0.0384],
        [ 1.5268, -1.6921, -0.9052, -1.0203, -0.5226, -0.7908,  0.2694, -1.4825,
         -0.4074],
        [ 1.8375,  2.9320,  2.4847,  1.7676, -0.0430, -0.8329,  0.8422, -3.7220,
         -1.4334],
        [ 0.9937, -1.7377, -0.8775,  2.1672,  2.2540, -0.4154,  0.3713, -3.9104,
          1.8341],
        [-1.0601, -1.7309, -0.5895, -2.9506,  2.2148,  2.8917, -0.3481,  1.0128,
         -4.0082]], device='cuda:0', grad_fn=<AddBackward0>)
>>> torch.logdet(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSgetrf( handle, m, n, dA, ldda, static_cast<float*>(dataPtr.get()), ipiv, info)`
>>> torch.det(b)
tensor(44733.0117, device='cuda:0', grad_fn=<DetLuBasedHelperBackward0>)
>>> torch.det(b)
tensor(44733.0117, device='cuda:0', grad_fn=<DetLuBasedHelperBackward0>)
>>> torch.det(b).log()
tensor(10.7085, device='cuda:0', grad_fn=<LogBackward0>)
>>> b.logdet()
tensor(10.7085, device='cuda:0', grad_fn=<LogdetBackward0>)
>>> torch.logdet(b)
tensor(10.7085, device='cuda:0', grad_fn=<LogdetBackward0>)
````
---
Motivated by this, I rewrote a new program to validate this.
```python
import torch

a = torch.randn(9, 9)
a = a.cuda()
a.requires_grad = True
print(a)
b= a+a
print(b)
try:
    print(torch.logdet(b))
    print('TRY-PART')
except:
    print(torch.logdet(b))
    print('EXCEPT-PART')
    print(torch.det(b))
    print(torch.det(b).log())
    print(b.logdet())
    print(torch.logdet(b))
```
THE OUTPUT:
```
tensor([[-0.6532,  0.5945,  1.1939, -2.1283, -1.1182,  0.4609, -1.0920,  1.1464,
         -2.6419],
        [-0.9333, -0.8353, -0.6816, -0.5360,  0.8662,  0.0058,  0.6089, -0.6144,
          1.5034],
        [-1.8466,  2.2491,  1.7100, -0.8738, -0.1071, -1.6346, -0.9584,  2.2552,
          1.0694],
        [ 0.1883,  0.7769,  0.5138, -1.3552, -0.6274,  1.5000,  0.0453, -0.8381,
          1.4051],
        [ 0.8021,  0.9736, -1.0843, -1.4855, -0.8864, -0.8064, -0.7061, -0.2059,
         -0.9930],
        [-1.4984,  2.1961,  0.4857, -0.1494,  0.7021,  0.3512,  0.3404,  0.5769,
         -0.7395],
        [ 0.8274, -1.4246,  1.1541, -0.5108, -0.8629, -0.4092, -0.3129,  0.4776,
          0.6271],
        [ 0.5916, -1.1913,  1.5054,  0.7082,  1.1280,  0.2026, -1.6208,  1.4507,
         -0.9418],
        [ 0.8289, -0.6652, -0.1179,  2.6689,  0.3450, -0.2769, -0.9009, -1.0749,
         -0.6171]], device='cuda:0', requires_grad=True)
tensor([[-1.3063,  1.1889,  2.3879, -4.2566, -2.2364,  0.9219, -2.1839,  2.2928,
         -5.2839],
        [-1.8666, -1.6706, -1.3632, -1.0719,  1.7323,  0.0115,  1.2178, -1.2288,
          3.0067],
        [-3.6932,  4.4981,  3.4200, -1.7475, -0.2142, -3.2692, -1.9168,  4.5104,
          2.1389],
        [ 0.3765,  1.5538,  1.0277, -2.7104, -1.2549,  3.0000,  0.0907, -1.6762,
          2.8102],
        [ 1.6041,  1.9472, -2.1685, -2.9710, -1.7728, -1.6128, -1.4121, -0.4118,
         -1.9860],
        [-2.9967,  4.3923,  0.9714, -0.2988,  1.4042,  0.7025,  0.6808,  1.1538,
         -1.4789],
        [ 1.6547, -2.8492,  2.3081, -1.0217, -1.7259, -0.8185, -0.6258,  0.9552,
          1.2542],
        [ 1.1832, -2.3826,  3.0107,  1.4165,  2.2561,  0.4051, -3.2417,  2.9013,
         -1.8836],
        [ 1.6578, -1.3303, -0.2359,  5.3378,  0.6900, -0.5538, -1.8019, -2.1498,
         -1.2343]], device='cuda:0', grad_fn=<AddBackward0>)
tensor(12.9578, device='cuda:0', grad_fn=<LogdetBackward0>)
EXCEPT-PART
tensor(424119.9688, device='cuda:0', grad_fn=<DetLuBasedHelperBackward0>)
tensor(12.9578, device='cuda:0', grad_fn=<LogBackward0>)
tensor(12.9578, device='cuda:0', grad_fn=<LogdetBackward0>)
tensor(12.9578, device='cuda:0', grad_fn=<LogdetBackward0>)
```
The code did not execute the 'try' part but executed successfully on the second attempt.
**WHY must it be executed twice to pass??? A BUG?**

### Versions

Collecting environment information...
PyTorch version: 1.10.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB
GPU 4: A100-SXM4-40GB
GPU 5: A100-SXM4-40GB
GPU 6: A100-SXM4-40GB
GPU 7: A100-SXM4-40GB

Nvidia driver version: 450.80.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.4
[pip3] torch==1.10.1+cu113
[pip3] torchvision==0.11.2
[conda] mkl                       2022.0.1                 pypi_0    pypi
[conda] mkl-fft                   1.3.1                    pypi_0    pypi
[conda] mkl-random                1.2.2                    pypi_0    pypi
[conda] numpy                     1.18.4                   pypi_0    pypi
[conda] torch                     1.10.1+cu113             pypi_0    pypi
[conda] torchvision               0.11.2                   pypi_0    pypi

cc @ngimel @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano