Repository: pytorch
Issue: Pytorch linalg test failure with cuda 11.6
Labels: module: cuda,triaged,module: linear algebra

### 🐛 Describe the bug

We recently introduced CUDA 11.6 to pytorch CI/CD pipeline. We are observing following failure on Linux : 

[linux-bionic-cuda11.6-py3.7-gcc7](https://github.com/pytorch/pytorch/runs/5859582068?check_suite_focus=true)

```
ERROR [0.017s]: test_matrix_rank_empty_cuda_complex128 (__main__.TestLinalgCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1780, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 389, in instantiated_test
    raise rte
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 376, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "test_linalg.py", line 3737, in test_matrix_rank_empty
    run_test(0, 3, batch)
  File "test_linalg.py", line 3723, in run_test
    self.assertEqual(matrix_rank(aHa), matrix_rank(aHa, hermitian=True))
torch._C._LinAlgError: torch.linalg.eigvalsh: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated eigenvalues (error code: 1).

======================================================================
ERROR [0.015s]: test_matrix_rank_empty_cuda_complex64 (__main__.TestLinalgCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1780, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 389, in instantiated_test
    raise rte
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 376, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "test_linalg.py", line 3737, in test_matrix_rank_empty
    run_test(0, 3, batch)
  File "test_linalg.py", line 3723, in run_test
    self.assertEqual(matrix_rank(aHa), matrix_rank(aHa, hermitian=True))
torch._C._LinAlgError: torch.linalg.eigvalsh: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated eigenvalues (error code: 1).

======================================================================
ERROR [0.015s]: test_matrix_rank_empty_cuda_float32 (__main__.TestLinalgCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1780, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 389, in instantiated_test
    raise rte
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 376, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "test_linalg.py", line 3737, in test_matrix_rank_empty
    run_test(0, 3, batch)
  File "test_linalg.py", line 3723, in run_test
    self.assertEqual(matrix_rank(aHa), matrix_rank(aHa, hermitian=True))
torch._C._LinAlgError: torch.linalg.eigvalsh: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated eigenvalues (error code: 1).

======================================================================
ERROR [0.016s]: test_matrix_rank_empty_cuda_float64 (__main__.TestLinalgCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1780, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 389, in instantiated_test
    raise rte
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 376, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 815, in dep_fn
    return fn(slf, *args, **kwargs)
  File "test_linalg.py", line 3737, in test_matrix_rank_empty
    run_test(0, 3, batch)
  File "test_linalg.py", line 3723, in run_test
    self.assertEqual(matrix_rank(aHa), matrix_rank(aHa, hermitian=True))
torch._C._LinAlgError: torch.linalg.eigvalsh: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated eigenvalues (error code: 1).

----------------------------------------------------------------------
Ran 757 tests in 328.426s

FAILED (errors=4, skipped=36, expected failures=12)

Generating XML reports...
Generated XML report: test-reports/python-unittest/test_linalg/TEST-TestLinalgCUDA-20220406234138.xml
Traceback (most recent call last):
  File "test/run_test.py", line 1056, in <module>
    main()
  File "test/run_test.py", line 1034, in main
    raise RuntimeError(err_message)
RuntimeError: test_linalg failed!
```

### Versions

CUDA 11.6
Pytorch master

cc @ngimel @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano