Repository: pytorch
Issue: `torch.pinverse` on CUDA tensors produces non-optimal output for certain type of (invertible) matrix on torch > 1.7.1 but not on torch <= 1.7.1
Labels: module: numerical-stability,module: cuda,triaged,module: linear algebra

### 🐛 Describe the bug

### Bug Description

`torch.pinverse / torch.linalg.pinv` produces non-optimal result at high probability when inversing matrix with structure `[A b | 0 1]`, especially when `||b|| >> ||A||`. In fact, the probability of failure raises significantly if `||b|| / ||A|| > 100`. 

This type of matrix is very common for coordinate transforms, and the triggering condition  `||b|| >> ||A||` is easily satisfied.

### Code to reproduce

```
import torch as th
import numpy as np

total_trials = 10000
failures_torch_cpu = 0
failures_torch_cuda = 0
failures_numpy = 0
failure_threshold = 1e-1

for i in range(total_trials):
    M = th.cat(
    [th.cat([th.diag(th.rand(3)*4), ((th.rand(3) - 0.5)*10000)[:, None]], dim=1), 
       th.tensor([[0, 0, 0, 1]], dtype=th.float)], dim=0)
    if (th.pinverse(M) @ M - th.eye(4).to(M)).abs().max() > failure_threshold:
        failures_torch_cpu += 1    
    M = M.cuda()
    if (th.pinverse(M) @ M - th.eye(4).to(M)).abs().max() > failure_threshold:
        failures_torch_cuda += 1
    if np.abs(np.linalg.pinv(M.cpu().numpy()) @ M.cpu().numpy() - np.eye(4)).max() > failure_threshold:
        failures_numpy += 1
        
print(f"""
Failures torch_cpu: {failures_torch_cpu / total_trials}.
Failures torch_cuda: {failures_torch_cuda / total_trials}.
Failures numpy: {failures_numpy / total_trials}.""")
```

Output for `torch==1.4.0, cuda==10.0`
```
Failures torch_cpu: 0.0018.
Failures torch_cuda: 0.0016.
Failures numpy: 0.0011.
```

Output for `torch==1.10.1, cuda==11.3`
```
Failures torch_cpu: 0.0017.
Failures torch_cuda: 0.5882.
Failures numpy: 0.0008.
```

Output for `torch==1.11.0, cuda==11.3`
```
Failures torch_cpu: 0.0014.
Failures torch_cuda: 0.5816.
Failures numpy: 0.0014.
```
### Workaround

- Use CPU implementation of `torch.linalg.pinv`
- Use `torch.linalg.inv` if you are sure that the matrix is invertible.
- Just use `numpy` instead.

### Versions

Env for `torch==1.11.0, cuda==11.3`
```
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.3.0-62-generic-x86_64-with-glibc2.10
Is CUDA available: N/A
CUDA runtime version: 11.3.109
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 2070
GPU 1: NVIDIA GeForce RTX 2070
GPU 2: NVIDIA GeForce RTX 2070
GPU 3: NVIDIA GeForce RTX 2070

Nvidia driver version: 495.46
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] Could not collect
[conda] blas                      1.0                         mkl    https://mirrors.infervision.com/anaconda/pkgs/main
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    https://mirrors.infervision.com/anaconda/pkgs/main
[conda] ffmpeg                    4.3                  hf484d3e_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] mkl                       2021.4.0           h06a4308_640    https://mirrors.infervision.com/anaconda/pkgs/main
[conda] mkl-service               2.4.0            py37h5e8e339_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] mkl_fft                   1.3.1            py37hd3c417c_0    https://mirrors.infervision.com/anaconda/pkgs/main
[conda] mkl_random                1.2.2            py37h219a48f_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] numpy                     1.21.5                   pypi_0    pypi
[conda] numpy-base                1.21.2           py37h79a1101_0    https://mirrors.infervision.com/anaconda/pkgs/main
[conda] pytorch                   1.11.0          py3.7_cuda11.3_cudnn8.2.0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] pytorch-mutex             1.0                        cuda    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torchaudio                0.11.0               py37_cu113    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torchvision               0.12.0               py37_cu113    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
```

Env for `torch==1.4.0, cuda==10.0`
```
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.3.0-62-generic-x86_64-with-glibc2.10
Is CUDA available: N/A
CUDA runtime version: 11.3.109
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 2070
GPU 1: NVIDIA GeForce RTX 2070
GPU 2: NVIDIA GeForce RTX 2070
GPU 3: NVIDIA GeForce RTX 2070

Nvidia driver version: 495.46
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] Could not collect
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.20.1                   pypi_0    pypi
[conda] numpyencoder              0.3.0                    pypi_0    pypi
[conda] torch                     1.4.0                    pypi_0    pypi
[conda] torchgeometry             0.1.2                    pypi_0    pypi
[conda] torchvision               0.5.0                    pypi_0    pypi
```


cc @ngimel @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano