Repository: pytorch
Issue: TestLinalgCPU.test_cholesky_solve_autograd_cpu_float64 fails on different random seed
Labels: module: autograd,triaged,module: linear algebra,module: correctness (silent)

## 🐛 Bug

`test_cholesky_solve_autograd_cpu_float64` looks like it gives potentially silently incorrect results depending on the random seed. It is possible that this is just a problem with numeric stability or that this is expected.

## To Reproduce

Before https://github.com/pytorch/pytorch/blob/6dda0363bb47fbfbceb242098f63b67560540584/test/test_linalg.py#L2144, add `torch.manual_seed(123)` (NB: does setting the seed mean that the behavior becomes the same across multiple machines?)

Then, run `python test/test_linalg.py -v TestLinalgCPU.test_cholesky_solve_autograd_cpu_float64`

The error message looks like:
<details><summary>CLICK ME</summary>
<p>

```
Traceback (most recent call last):
  File "/raid/rzou/pt/whiteboard/torch/testing/_internal/common_device_type.py", line 286, in instantiated_test
    raise rte
  File "/raid/rzou/pt/whiteboard/torch/testing/_internal/common_device_type.py", line 281, in instantiated_test
    result = test_fn(self, *args)
  File "/raid/rzou/pt/whiteboard/torch/testing/_internal/common_device_type.py", line 572, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/raid/rzou/pt/whiteboard/torch/testing/_internal/common_device_type.py", line 572, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test/test_linalg.py", line 2137, in test_cholesky_solve_autograd
    run_test(a_size, b_size, upper)
  File "test/test_linalg.py", line 2132, in run_test
    gradgradcheck(func, [root, b, upper], atol=1e-3, check_batched_grad=False)
  File "/raid/rzou/pt/whiteboard/torch/autograd/gradcheck.py", line 677, in gradgradcheck
    check_grad_dtypes=check_grad_dtypes, check_batched_grad=check_batched_grad)
  File "/raid/rzou/pt/whiteboard/torch/autograd/gradcheck.py", line 468, in gradcheck
    checkIfNumericalAnalyticAreClose(a, n, j)
  File "/raid/rzou/pt/whiteboard/torch/autograd/gradcheck.py", line 450, in checkIfNumericalAnalyticAreClose
    'numerical:%s\nanalytical:%s\n' % (i, j, n, a))
  File "/raid/rzou/pt/whiteboard/torch/autograd/gradcheck.py", line 367, in fail_test
    raise RuntimeError(msg)
RuntimeError: Jacobian mismatch for output 0 with respect to input 1,
numerical:tensor([[-7.3962e+05,  0.0000e+00,  0.0000e+00,  1.7113e+05,  2.4795e+07,
          0.0000e+00, -1.5407e+04, -2.2333e+06,  3.4593e+07],
        [-2.3094e+06,  0.0000e+00,  0.0000e+00,  5.3445e+05,  9.8830e+06,
          0.0000e+00, -4.8144e+04, -8.9023e+05,  1.3988e+07],
        [ 6.1956e+06,  0.0000e+00,  0.0000e+00, -1.4340e+06, -1.9483e+07,
          0.0000e+00,  1.2917e+05,  1.7551e+06, -2.8155e+07],
        [ 3.8187e+06,  0.0000e+00,  0.0000e+00, -8.8377e+05, -9.3427e+06,
          0.0000e+00,  7.9628e+04,  8.4198e+05, -1.4833e+07],
        [-3.1341e+05,  0.0000e+00,  0.0000e+00,  7.2549e+04, -5.8620e+06,
          0.0000e+00, -6.5351e+03,  5.2795e+05, -8.0050e+06],
        [ 3.3861e+05,  0.0000e+00,  0.0000e+00, -7.8376e+04, -2.3372e+06,
          0.0000e+00,  7.0592e+03,  2.1051e+05, -3.2368e+06],
        [-1.0398e+06,  0.0000e+00,  0.0000e+00,  2.4059e+05,  4.6096e+06,
          0.0000e+00, -2.1672e+04, -4.1522e+05,  6.5151e+06],
        [-6.7630e+05,  0.0000e+00,  0.0000e+00,  1.5651e+05,  2.2151e+06,
          0.0000e+00, -1.4094e+04, -1.9962e+05,  3.4325e+06],
        [ 2.8172e+04,  0.0000e+00,  0.0000e+00, -6.5597e+03,  2.6236e+05,
          0.0000e+00,  5.9254e+02, -2.3732e+04,  7.2410e+05],
        [-3.0425e+04,  0.0000e+00,  0.0000e+00,  7.1140e+03,  1.0311e+05,
          0.0000e+00, -6.3604e+02, -9.3273e+03,  2.9279e+05],
        [ 9.3690e+04,  0.0000e+00,  0.0000e+00, -2.1672e+04, -1.9905e+05,
          0.0000e+00,  1.9576e+03,  1.8006e+04, -5.8934e+05],
        [ 6.0737e+04,  0.0000e+00,  0.0000e+00, -1.4121e+04, -8.5730e+04,
          0.0000e+00,  1.2785e+03,  7.7547e+03, -3.1050e+05]],
       dtype=torch.float64)
analytical:tensor([[-7.3935e+05,  0.0000e+00,  0.0000e+00,  1.7113e+05,  2.4795e+07,
          0.0000e+00, -1.5409e+04, -2.2333e+06,  3.4593e+07],
        [-2.3093e+06,  0.0000e+00,  0.0000e+00,  5.3447e+05,  9.8830e+06,
          0.0000e+00, -4.8144e+04, -8.9023e+05,  1.3988e+07],
        [ 6.1957e+06,  0.0000e+00,  0.0000e+00, -1.4340e+06, -1.9483e+07,
          0.0000e+00,  1.2917e+05,  1.7551e+06, -2.8155e+07],
        [ 3.8186e+06,  0.0000e+00,  0.0000e+00, -8.8380e+05, -9.3427e+06,
          0.0000e+00,  7.9627e+04,  8.4198e+05, -1.4833e+07],
        [-3.1351e+05, -0.0000e+00, -0.0000e+00,  7.2562e+04, -5.8620e+06,
         -0.0000e+00, -6.5349e+03,  5.2795e+05, -8.0050e+06],
        [ 3.3853e+05, -0.0000e+00, -0.0000e+00, -7.8352e+04, -2.3372e+06,
         -0.0000e+00,  7.0565e+03,  2.1051e+05, -3.2368e+06],
        [-1.0396e+06, -0.0000e+00, -0.0000e+00,  2.4062e+05,  4.6096e+06,
         -0.0000e+00, -2.1670e+04, -4.1522e+05,  6.5151e+06],
        [-6.7620e+05, -0.0000e+00, -0.0000e+00,  1.5650e+05,  2.2151e+06,
         -0.0000e+00, -1.4095e+04, -1.9962e+05,  3.4325e+06],
        [ 2.8240e+04,  0.0000e+00,  0.0000e+00, -6.5349e+03,  2.6235e+05,
          0.0000e+00,  5.9113e+02, -2.3732e+04,  7.2410e+05],
        [-3.0494e+04,  0.0000e+00,  0.0000e+00,  7.0565e+03,  1.0311e+05,
          0.0000e+00, -6.3831e+02, -9.3273e+03,  2.9279e+05],
        [ 9.3647e+04,  0.0000e+00,  0.0000e+00, -2.1670e+04, -1.9905e+05,
          0.0000e+00,  1.9602e+03,  1.8005e+04, -5.8934e+05],
        [ 6.0910e+04,  0.0000e+00,  0.0000e+00, -1.4095e+04, -8.5729e+04,
          0.0000e+00,  1.2750e+03,  7.7548e+03, -3.1050e+05]],
       dtype=torch.float64)
```

</p>
</details>

Some of the values are off by a lot, e.g. 6.0737e+04 vs 6.0910e+04. This is especially concerning because the precision is float64.

## Expected behavior

Is this behavior expected? If so, we should probably tweak the tolerances or set the manual seed in the test so that some change in randomness doesn't cause the test to be flaky.

## Environment

master

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @albanD @gqchen @pearu @nikitaved @soulitzer @jianyuh @mruberry @heitorschueroff @walterddr @IvanYashchuk