Repository: pytorch
Issue: `(svd|pca)_lowrank`: backward is unstable when for a matrix `A`, the parameter `q` is set to a value `q > rank(A)`.
Labels: module: numerical-stability,module: autograd,triaged,module: linear algebra,Stale

## 🐛 Bug

As per title. `q` is supposed to slightly overestimate the rank of the input, which renders the backward of these low-rank methods useless in most real-life applications. 

## To Reproduce

```python
In [1]: import torch

In [2]: x = torch.rand(30, 1, dtype=torch.double, requires_grad=True)

In [3]: y = torch.rand(30, 1, dtype=torch.double, requires_grad=True)

In [4]: def run_gradcheck(q=1):
   ...:     torch.autograd.gradcheck(lambda a, b: torch.svd_lowrank(a @ b.mT, q=q), [x, y])
   ...: 

In [5]: run_gradcheck()

In [6]: run_gradcheck(2)
---------------------------------------------------------------------------
GradcheckError                            Traceback (most recent call last)
<ipython-input-6-d439ddc043f2> in <module>
----> 1 run_gradcheck(2)

<ipython-input-4-4ebcbeade42d> in run_gradcheck(q)
      1 def run_gradcheck(q=1):
----> 2     torch.autograd.gradcheck(lambda a, b: torch.svd_lowrank(a @ b.mT, q=q), [x, y])
      3 

~/git/Quansight/pytorch/torch/autograd/gradcheck.py in gradcheck(func, inputs, eps, atol, rtol, raise_exception, check_sparse_nnz, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad, check_backward_ad, fast_mode)
   1333             return False
   1334     else:
-> 1335         return _gradcheck_helper(**args)
   1336 
   1337 

~/git/Quansight/pytorch/torch/autograd/gradcheck.py in _gradcheck_helper(func, inputs, eps, atol, rtol, check_sparse_nnz, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad, check_backward_ad, fast_mode)
   1347 
   1348     gradcheck_fn = _fast_gradcheck if fast_mode else _slow_gradcheck
-> 1349     _gradcheck_real_imag(gradcheck_fn, func, func_out, tupled_inputs, outputs, eps,
   1350                          rtol, atol, check_grad_dtypes, check_forward_ad=check_forward_ad,
   1351                          check_backward_ad=check_backward_ad, nondet_tol=nondet_tol)

~/git/Quansight/pytorch/torch/autograd/gradcheck.py in _gradcheck_real_imag(gradcheck_fn, func, func_out, tupled_inputs, outputs, eps, rtol, atol, check_grad_dtypes, check_forward_ad, check_backward_ad, nondet_tol)
   1003                          rtol, atol, check_grad_dtypes, nondet_tol, complex_indices=complex_out_indices)
   1004         else:
-> 1005             gradcheck_fn(func, func_out, tupled_inputs, outputs, eps,
   1006                          rtol, atol, check_grad_dtypes, nondet_tol)
   1007 

~/git/Quansight/pytorch/torch/autograd/gradcheck.py in _slow_gradcheck(func, func_out, tupled_inputs, outputs, eps, rtol, atol, check_grad_dtypes, nondet_tol, use_forward_ad, complex_indices, test_imag)
   1052             for j, (a, n) in enumerate(zip(analytical, numerical[i])):
   1053                 if not _allclose_with_type_promotion(a, n.to(a.device), rtol, atol):
-> 1054                     raise GradcheckError(_get_notallclose_msg(a, n, i, j, complex_indices, test_imag))
   1055 
   1056     return True

GradcheckError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[-2.6942e-01,  1.8734e+05,  9.7665e-03,  ..., -8.7014e+04,
          1.3115e-02, -1.0406e+05],
        [ 9.7665e-03, -8.3856e+04, -2.6970e-01,  ..., -3.2701e+04,
          1.2926e-02, -1.5542e+05],
        [ 8.6690e-03, -2.0546e+05,  8.5441e-03,  ...,  6.6438e+04,
          1.1474e-02,  2.8275e+05],
        ...,
        [ 1.3242e-03, -6.1347e+03,  1.3051e-03,  ..., -4.4975e+04,
          1.7526e-03,  9.5841e+04],
        [ 3.4860e-03,  2.7576e+05,  3.4358e-03,  ...,  1.7474e+04,
          4.6138e-03, -3.8359e+04],
        [ 1.3115e-02,  1.6119e+05,  1.2926e-02,  ..., -4.3518e+04,
         -2.6197e-01,  7.0259e+04]], dtype=torch.float64)
analytical:tensor([[-2.6942e-01, -8.5772e+02,  9.7665e-03,  ...,  6.2188e+00,
          1.3115e-02, -1.8000e+01],
        [ 9.7665e-03, -1.5902e+03, -2.6970e-01,  ...,  1.1844e+01,
          1.2926e-02, -3.4000e+01],
        [ 8.6690e-03,  6.8769e+02,  8.5441e-03,  ..., -4.9531e+00,
          1.1474e-02,  1.5500e+01],
        ...,
        [ 1.3242e-03,  3.7783e+02,  1.3051e-03,  ..., -2.7578e+00,
          1.7526e-03,  8.2500e+00],
        [ 3.4860e-03,  1.0674e+02,  3.4358e-03,  ..., -5.9180e-01,
          4.6138e-03,  2.1299e+00],
        [ 1.3115e-02,  7.4394e+02,  1.2926e-02,  ..., -5.5215e+00,
         -2.6197e-01,  1.5719e+01]], dtype=torch.float64)

```

## Additional context

The issues is with the backward for the SVD algorithm as the final step for the approximation algorithm behind `svd_lowrank`. It has a limitation of not being stable with repeated and or zero singular values. The issue of (repeated) zero singular values, however, could be fixed with a rank-revealing SVD method implementation.


cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @jianyuh @mruberry @walterddr @IvanYashchuk @xwang233