Repository: pytorch
Issue: pytorch 1.8.2  cuda test errors
Labels: module: build,module: autograd,module: cuda,triaged,module: linear algebra

Hi,

System is Ubuntu 20.04
python 3.8.10
CUDA 11.0
cudnn 8
GPU Nvidia GTX1070

I built pytorch with these commands
```
export BUILD_ONNX_PYTHON=1 USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1 USE_TENSORRT=1 USE_OPENCV=1 USE_FFMPEG=1 USE_LMDB=1 GPU_ARCH=sm_61
 git clone https://github.com/pytorch/pytorch.git
cd pytorch
git checkout v1.8.2
git submodule sync
git submodule update --init --recursive
 python setup.py install
```
Build was successful but running test produced these errors

```
ERROR: test_lu_square_many_batches_no_info_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 874, in wrapper
    method(*args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 295, in instantiated_test
    raise rte
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 290, in instantiated_test
    result = test_fn(self, *args)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test_autograd.py", line 5176, in do_test
    check(name)
  File "test_autograd.py", line 5082, in check
    run_grad_and_gradgrad_checks(self, name, test_name, fn,
  File "test_autograd.py", line 4959, in run_grad_and_gradgrad_checks
    test_case.assertTrue(gradgradcheck(apply_method, input_variables,
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1925, in gradgradcheck
    return torch.autograd.gradgradcheck(fn, inputs, grad_outputs, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 674, in gradgradcheck
    return gradcheck(
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 468, in gradcheck
    checkIfNumericalAnalyticAreClose(a, n, j)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 449, in checkIfNumericalAnalyticAreClose
    return fail_test(error_str + 'Jacobian mismatch for output %d with respect to input %d,\n'
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 367, in fail_test
    raise RuntimeError(msg)
RuntimeError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[ 3.6142e+00,  7.0091e+00,  5.3443e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 7.0091e+00,  1.1945e+01,  9.4077e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 5.3443e+00,  9.4077e+00,  7.4169e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        ...,
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  1.4775e+00,
          1.3568e+00, -9.9772e-02],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  1.3568e+00,
          1.2459e+00, -9.1998e-02],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -9.9772e-02,
         -9.1998e-02, -1.1102e-10]], device='cuda:0')
analytical:tensor([[ 3.6142,  7.0091,  5.3443,  ...,  0.0000,  0.0000,  0.0000],
        [ 7.0091, 11.9455,  9.4077,  ...,  0.0000,  0.0000,  0.0000],
        [ 5.3443,  9.4077,  7.4169,  ...,  0.0000,  0.0000,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000,  ...,  1.4775,  1.3568, -0.0998],
        [ 0.0000,  0.0000,  0.0000,  ...,  1.3568,  1.2459, -0.0920],
        [ 0.0000,  0.0000,  0.0000,  ..., -0.0998, -0.0920,  0.0000]],
       device='cuda:0')


======================================================================
ERROR: test_lu_square_many_batches_with_info_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 874, in wrapper
    method(*args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 295, in instantiated_test
    raise rte
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 290, in instantiated_test
    result = test_fn(self, *args)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test_autograd.py", line 5176, in do_test
    check(name)
  File "test_autograd.py", line 5082, in check
    run_grad_and_gradgrad_checks(self, name, test_name, fn,
  File "test_autograd.py", line 4947, in run_grad_and_gradgrad_checks
    test_case.assertTrue(gradcheck(apply_method, input_variables, eps=1e-6, atol=PRECISION,
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1911, in gradcheck
    return torch.autograd.gradcheck(fn, inputs, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 468, in gradcheck
    checkIfNumericalAnalyticAreClose(a, n, j)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 449, in checkIfNumericalAnalyticAreClose
    return fail_test(error_str + 'Jacobian mismatch for output %d with respect to input %d,\n'
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 367, in fail_test
    raise RuntimeError(msg)
RuntimeError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000, -0.4702, -0.3404],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000, -0.4336, -0.2948],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.3413]],
       device='cuda:0')
analytical:tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000, -0.4702, -0.3404],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000, -0.4336, -0.2948],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.3413]],
       device='cuda:0')


======================================================================
ERROR: test_qr_tall_many_batched_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 874, in wrapper
    method(*args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 295, in instantiated_test
    raise rte
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 290, in instantiated_test
    result = test_fn(self, *args)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test_autograd.py", line 5176, in do_test
    check(name)
  File "test_autograd.py", line 5082, in check
    run_grad_and_gradgrad_checks(self, name, test_name, fn,
  File "test_autograd.py", line 4959, in run_grad_and_gradgrad_checks
    test_case.assertTrue(gradgradcheck(apply_method, input_variables,
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1925, in gradgradcheck
    return torch.autograd.gradgradcheck(fn, inputs, grad_outputs, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 674, in gradgradcheck
    return gradcheck(
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 495, in gradcheck
    test_batched_grad(fail_test, tupled_inputs, o, j)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 279, in test_batched_grad
    return fail_test(get_failed_batched_grad_test_msg(output_idx, input_idx, res, exp))
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 367, in fail_test
    raise RuntimeError(msg)
RuntimeError: For output 2 and input 0:

gradcheck or gradgradcheck failed while testing batched gradient computation.
This could have been invoked in a number of ways (via a test that calls
gradcheck/gradgradcheck directly or via an autogenerated test).

If you are adding a new operator, please file an issue and then use one of the
workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck.
If the test
- manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck
  with `check_batched_grad=False` as a keyword argument.
- is OpInfo-based (e.g., in test_ops.py), then modify the OpInfo for the test
  to have `check_batched_grad=False` and/or `check_batched_gradgrad=False`.
- is common_method_invocations-based, then add your test to the denylist
  EXCLUDE_BATCHED_GRAD_TESTS in test_autograd.py

If you're modifying an existing operator that supports batched grad computation,
or wish to make a new operator work with batched grad computation, please read
the following.

To compute batched grads (e.g., jacobians, hessians), we vmap over the backward
computation. The most common failure case is if there is a 'vmap-incompatible
operation' in the backward pass. Please see
NOTE: [How to write vmap-compatible backward formulas]
in the codebase for an explanation of how to fix this.

Got:
tensor([[[[[  2.9622,   0.9807,  -0.3569],
           [ -0.6038,   0.8646,   0.3064],
           [ -2.8353,   1.8364,   0.5928],
           [ -0.0796,  -1.2900,  -0.2605],
           [ -0.9634,  -0.9452,  -0.1412]],

          [[ -0.9675,   6.4359,  -0.9693],
           [  1.0163,   1.0250,   1.3283],
           [  2.2968,   1.1729,  -0.8792],
           [  6.0633,  -1.6275,  -0.8841],
           [  0.0223,   6.9330,  -1.3414]]],


         [[[ -3.6968,  -0.5336,  -1.5816],
           [ -1.2205,  -0.3288,   2.0625],
           [ -3.8076,  -0.7206,  -1.1479],
           [ -2.4978,  -0.7226,   2.3435],
           [ -5.2436,   3.0036,  -3.5995]],

          [[  3.2744,  -0.7005,  -1.8554],
           [ -1.1138,  -3.9561,  -4.1945],
           [  4.5882,  -0.4164,   2.0327],
           [ -4.9854,  -1.8817,   2.7343],
           [  5.8144,   1.5108,   0.8854]]],


         [[[ -1.1379,   1.4698,  -2.9472],
           [ -2.2710,   1.8941,   0.1217],
           [  0.6506,   1.1031,  -1.4135],
           [  1.8249,   1.0863,   1.3407],
           [ -1.7504,  -2.2775,   2.3955]],

          [[ -4.5792,   1.1060,  -1.6547],
           [ -1.0279,  -0.3496,   0.4365],
           [ -2.0371,  -0.7384,   0.1931],
           [  0.4660,   0.4655,  -0.6202],
           [  1.8433,  -1.5225,   0.5121]]]],



        [[[[ -1.3276,  -1.9036,   2.7275],
           [-13.3521,  10.0619, -20.2156],
           [ -5.9001,   1.3122,  -5.9610],
           [  1.4125,  -3.6834,   5.9334],
           [ -6.1040,   3.4439,  -7.5539]],

          [[  7.4349,   5.3774,  -1.9584],
           [  7.2063,   6.5611,   1.9743],
           [  5.4678,   5.0245,  -1.2843],
           [-10.3261,  -8.1677,   0.6731],
           [  1.4354,  -0.4733,  -1.7204]]],


         [[[ -0.2550,   8.9270,   5.3065],
           [  0.6546,   8.0141,   2.8593],
           [  1.0570,  -2.8252,  -2.2300],
           [  0.4876,  -7.1121,  -1.9424],
           [  2.2354,   2.9225,   2.6713]],

          [[  1.3293,   5.3037,   2.9897],
           [  1.0291,   2.9323,   3.0617],
           [  0.3262,   2.8591,   0.3443],
           [  0.9064,  -2.1562,   3.3413],
           [ -2.7875,   4.7553,   0.4868]]],


         [[[  0.1934,  -0.9530,   0.0825],
           [  1.5300,  -0.7924,   0.5422],
           [  0.0530,   0.1670,  -0.6090],
           [ -0.1249,   1.3006,  -0.4560],
           [  0.1508,  -1.8492,   0.3713]],

          [[  0.5739,  -0.6955,  -0.6562],
           [ -0.8399,  -1.1456,  -0.9745],
           [ -0.8026,   1.3733,   1.5225],
           [ -0.6126,   0.9344,   1.6060],
           [ -1.0505,  -0.1363,  -0.5126]]]]], device='cuda:0')

Expected:
tensor([[[[[  2.6125,  -0.8313,   0.7901],
           [ -1.9829,  -0.5354,  -0.6781],
           [ -5.2594,  -1.1557,  -1.3123],
           [  1.6310,   0.8614,   0.5766],
           [  0.3461,   0.6469,   0.3126]],

          [[ -8.8698,  -9.1198,  -1.1540],
           [ -0.6994,   0.6990,   1.5814],
           [ -3.7217,  -2.6815,  -1.0467],
           [ -2.2170,   0.6676,  -1.0525],
           [-10.4561, -10.2554,  -1.5969]]],


         [[[  1.3291,   3.7505,   1.5157],
           [  1.2762,  -3.6392,  -1.9765],
           [  1.4853,   3.1292,   1.1001],
           [  1.9591,  -3.7087,  -2.2458],
           [  2.4658,   3.3830,   3.4494]],

          [[ -1.6708,   2.9712,   1.9563],
           [ -1.3315,  10.1763,   4.4227],
           [ -1.8747,  -1.5289,  -2.1433],
           [  1.3829,  -0.1297,  -2.8831],
           [ -1.6657,  -3.1333,  -0.9336]]],


         [[[ -0.3644,   1.6526,  -1.2125],
           [ -0.5189,   0.8653,   0.0500],
           [  0.3102,   0.9828,  -0.5815],
           [  0.8921,   0.0863,   0.5516],
           [ -0.8169,  -1.8603,   0.9855]],

          [[ -1.3912,  -0.4239,  -1.2588],
           [ -0.6579,   0.1337,   0.3321],
           [ -0.9863,   0.2796,   0.1469],
           [  0.4792,  -0.1781,  -0.4718],
           [  0.6919,   0.5769,   0.3896]]]],



        [[[[ -1.3276,  -1.9036,   2.7275],
           [-13.3521,  10.0619, -20.2156],
           [ -5.9001,   1.3122,  -5.9610],
           [  1.4125,  -3.6834,   5.9334],
           [ -6.1040,   3.4439,  -7.5539]],

          [[  7.4349,   5.3774,  -1.9584],
           [  7.2063,   6.5611,   1.9743],
           [  5.4678,   5.0245,  -1.2843],
           [-10.3261,  -8.1677,   0.6731],
           [  1.4354,  -0.4733,  -1.7204]]],


         [[[ -0.2550,   8.9270,   5.3065],
           [  0.6546,   8.0141,   2.8593],
           [  1.0570,  -2.8252,  -2.2300],
           [  0.4876,  -7.1121,  -1.9424],
           [  2.2354,   2.9225,   2.6713]],

          [[  1.3293,   5.3037,   2.9897],
           [  1.0291,   2.9323,   3.0617],
           [  0.3262,   2.8591,   0.3443],
           [  0.9064,  -2.1562,   3.3413],
           [ -2.7875,   4.7553,   0.4868]]],


         [[[  0.1934,  -0.9530,   0.0825],
           [  1.5300,  -0.7924,   0.5422],
           [  0.0530,   0.1670,  -0.6090],
           [ -0.1249,   1.3006,  -0.4560],
           [  0.1508,  -1.8492,   0.3713]],

          [[  0.5739,  -0.6955,  -0.6562],
           [ -0.8399,  -1.1456,  -0.9745],
           [ -0.8026,   1.3733,   1.5225],
           [ -0.6126,   0.9344,   1.6060],
           [ -1.0505,  -0.1363,  -0.5126]]]]], device='cuda:0')

======================================================================
FAIL: test_lu_square_batch_with_info_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 874, in wrapper
    method(*args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 290, in instantiated_test
    result = test_fn(self, *args)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 581, in dep_fn
    return fn(slf, device, *args, **kwargs)
  File "test_autograd.py", line 5176, in do_test
    check(name)
  File "test_autograd.py", line 5073, in check
    self.assertEqual(unpack_variables(output_variable), output_tensor)
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1198, in assertEqual
    self.assertEqual(x_, y_, atol=atol, rtol=rtol, msg=msg,
  File "/home/bernard/opt/python38/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1165, in assertEqual
    super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
AssertionError: False is not true : Tensors failed to compare as equal!With rtol=1e-07 and atol=1e-07, found 75 element(s) (out of 75) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 3.392036083082407 (-2.45488986289944 vs. 0.937146220182967), which occurred at index (2, 3, 3).

----------------------------------------------------------------------
Ran 2715 tests in 1345.736s

FAILED (failures=1, errors=3, skipped=23, expected failures=1)
Traceback (most recent call last):
  File "test/run_test.py", line 926, in <module>
    main()
  File "test/run_test.py", line 905, in main
    raise RuntimeError(err_message)
RuntimeError: test_autograd failed!

```

Some of it is the same as https://github.com/pytorch/pytorch/issues/59855, but there seem to be more failures this time.

cc @malfet @seemethere @walterddr @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @ngimel @jianyuh @mruberry @heitorschueroff @IvanYashchuk @xwang233