Repository: pytorch
Issue: conv3d padding=`same` gradgradcheck fails on CUDA
Labels: needs reproduction,module: autograd,module: convolution,triaged,module: determinism

To replicate, remove the `if torch.device(device).type != 'cuda'` condition in `test_conv3d_same_padding_backward` and run CI

Note: I couldn't actually replicate these locally
```
======================================================================
ERROR [0.068s]: test_conv3d_same_padding_backward_cuda (__main__.TestNNDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1566, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 381, in instantiated_test`
    raise rte
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 376, in instantiated_test
    result = test(self, **param_kwargs)
  File "test_nn.py", line 13430, in test_conv3d_same_padding_backward
    check_fwd_over_rev=True, nondet_tol=1e-5, atol=1e-4)
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 2977, in gradgradcheck
    return torch.autograd.gradgradcheck(fn, inputs, grad_outputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1550, in gradgradcheck
    check_forward_ad=check_fwd_over_rev, check_backward_ad=check_rev_over_rev)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1400, in gradcheck
    return _gradcheck_helper(**args)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1417, in _gradcheck_helper
    check_undefined_grad=check_undefined_grad)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1089, in _gradcheck_real_imag
    _test_undefined_forward_mode(func, outputs, tupled_inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 927, in _test_undefined_forward_mode
    " Got: ", res1, " but expected: ", res2)
torch.autograd.gradcheck.GradcheckError: ('Mismatch in tangent values for output with index: ', 0, ' when input: ', tensor([[[[[0.85347, 0.32417, 0.92350, 0.74723, 0.62705, 0.71730, 0.80550,
            0.12902, 0.61274, 0.88229, 0.02407, 0.79346],
           [0.24910, 0.52173, 0.05480, 0.08712, 0.63187, 0.86262, 0.20201,
            0.07088, 0.27600, 0.96250, 0.52568, 0.56010],
           [0.07013, 0.52118, 0.91096, 0.44277, 0.66309, 0.66927, 0.61083,
            0.53504, 0.56099, 0.14079, 0.55776, 0.48921],
           [0.23373, 0.30994, 0.91836, 0.38548, 0.65276, 0.53187, 0.35904,
            0.93752, 0.57353, 0.26192, 0.32132, 0.96646],
           [0.42355, 0.22348, 0.69466, 0.18496, 0.66242, 0.40323, 0.13950,
            0.97400, 0.98626, 0.62311, 0.76556, 0.44082],
           [0.37164, 0.89845, 0.33609, 0.65906, 0.30702, 0.63686, 0.03229,
            0.22066, 0.03233, 0.45583, 0.39696, 0.72413],
           [0.80862, 0.57028, 0.59517, 0.06096, 0.78281, 0.43785, 0.40212,
            0.35835, 0.44497, 0.56136, 0.05265, 0.94099],
           [0.51706, 0.96307, 0.30411, 0.23571, 0.79196, 0.53767, 0.41476,
            0.64928, 0.87583, 0.79490, 0.37542, 0.63998],
           [0.43485, 0.21873, 0.99407, 0.92236, 0.26950, 0.52603, 0.65273,
            0.09113, 0.01691, 0.78774, 0.18691, 0.61338],
           [0.60991, 0.87659, 0.30172, 0.59689, 0.00461, 0.98603, 0.24432,
            0.50340, 0.14649, 0.51435, 0.88475, 0.13425],
           [0.82042, 0.06272, 0.34525, 0.50688, 0.15624, 0.76729, 0.23308,
            0.96281, 0.57974, 0.40961, 0.02813, 0.91491]]]]], device='cuda:0',
       requires_grad=True), ' has an undefined tangent value. ', ' Got: ', tensor([[[[[ 1.68295, -0.38621,  1.10320, -0.69561, -0.03474, -0.18586,
            -0.24399,  0.36020,  0.02305, -0.61639,  0.45284, -0.48569],
           [-0.16269,  1.33667, -0.53159,  0.80608, -0.20259,  2.08437,
             0.71256,  2.25388,  1.63294,  1.13542,  1.21990,  1.08208],
           [ 1.47756, -0.91170,  0.53310, -1.37292, -0.07164,  0.10335,
            -0.62000, -0.03664, -0.19522,  0.02120,  0.24184,  1.47398],
           [-0.93896,  0.87402, -0.49792,  1.24801,  1.29966,  2.70651,
             1.43205,  0.63347,  1.85086,  0.33770,  1.83747,  0.22465],
           [ 0.90719, -0.11004,  0.48741, -0.21199, -0.64756,  1.77254,
            -0.11713,  1.22362,  1.41854,  2.58232,  0.84034,  1.75236],
           [-0.47378,  1.35394,  0.15682,  2.11509,  0.45029,  1.20467,
             0.93702, -0.02698,  1.73497,  0.06297,  0.69825, -0.28669],
           [-0.08635, -0.27524, -0.97766,  0.43028, -0.26773,  1.49096,
             1.18282,  1.27320,  1.06322,  1.84124,  0.09761,  0.69117],
           [ 0.03052,  1.01699,  1.03550,  1.20171,  0.99675,  0.53350,
             1.37765, -0.28267,  0.52546, -0.78914,  0.38847, -0.31154],
           [-2.01115, -0.76533, -0.98516, -0.85194, -0.14157, -0.52706,
             0.68569,  0.04554,  1.38594,  1.49958,  1.48046,  1.50072],
           [ 3.40671, -0.20024,  3.24647,  0.63427,  1.56040,  1.57964,
             0.93242,  0.19733,  0.26865, -0.35869,  0.46059, -0.67451],
           [-0.56967, -0.66194, -0.25630, -0.41094, -0.27457, -0.45702,
             0.80764,  0.25628,  1.09684,  0.98682,  0.58633,  0.42737]]]]],
       device='cuda:0', grad_fn=<AddBackward0>), ' but expected: ', tensor([[[[[ 1.68295, -0.38621,  1.10320, -0.69561, -0.03474, -0.18586,
            -0.24399,  0.36020,  0.02305, -0.61639,  0.45284, -0.48569],
           [-0.16269,  1.33667, -0.53159,  0.80608, -0.20259,  2.08437,
             0.71256,  2.25388,  1.63294,  1.13542,  1.21990,  1.08208],
           [ 1.47756, -0.91170,  0.53310, -1.37292, -0.07164,  0.10335,
            -0.62000, -0.03664, -0.19522,  0.02120,  0.24184,  1.47398],
           [-0.93896,  0.87402, -0.49792,  1.24801,  1.29966,  2.70651,
             1.43205,  0.63347,  1.85086,  0.33770,  1.83747,  0.22465],
           [ 0.90719, -0.11004,  0.48741, -0.21199, -0.64756,  1.77254,
            -0.11713,  1.22362,  1.41854,  2.58232,  0.84034,  1.75236],
           [-0.47378,  1.35394,  0.15682,  2.11509,  0.45029,  1.20467,
             0.93702, -0.02698,  1.73497,  0.06297,  0.69825, -0.28669],
           [-0.08635, -0.27524, -0.97766,  0.43028, -0.26773,  1.49096,
             1.18282,  1.27320,  1.06322,  1.84124,  0.09761,  0.69117],
           [ 0.03052,  1.01699,  1.03550,  1.20171,  0.99675,  0.53350,
             1.37765, -0.28267,  0.52546, -0.78914,  0.38847, -0.31154],
           [-2.01115, -0.76533, -0.98516, -0.85194, -0.14157, -0.52706,
             0.68569,  0.04554,  1.38594,  1.49958,  1.48046,  1.50072],
           [ 3.40671, -0.20024,  3.24647,  0.63427,  1.56040,  1.57964,
             0.93242,  0.19733,  0.26865, -0.35869,  0.46059, -0.67451],
           [-0.56967, -0.66194, -0.25630, -0.41094, -0.27457, -0.45702,
             0.80764,  0.25628,  1.09684,  0.98682,  0.58633,  0.42737]]]]],
       device='cuda:0', grad_fn=<AddBackward0>))
```

```
======================================================================
ERROR [0.023s]: test_conv3d_same_padding_backward_cuda (__main__.TestNNDeviceTypeCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1566, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 381, in instantiated_test
    raise rte
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 376, in instantiated_test
    result = test(self, **param_kwargs)
  File "test_nn.py", line 13432, in test_conv3d_same_padding_backward
    check_fwd_over_rev=True)
  File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 2977, in gradgradcheck
    return torch.autograd.gradgradcheck(fn, inputs, grad_outputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1550, in gradgradcheck
    check_forward_ad=check_fwd_over_rev, check_backward_ad=check_rev_over_rev)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1400, in gradcheck
    return _gradcheck_helper(**args)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1417, in _gradcheck_helper
    check_undefined_grad=check_undefined_grad)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1062, in _gradcheck_real_imag
    rtol, atol, check_grad_dtypes, nondet_tol)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 1291, in _fast_gradcheck
    analytical_vJu = _get_analytical_vJu_backward_mode(inputs, outputs, nondet_tol, check_grad_dtypes, all_v, all_u_dense)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 556, in _get_analytical_vJu_backward_mode
    fast_mode=True, v=v)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 548, in _check_analytical_jacobian_attributes
    FAILED_NONDET_MSG)
torch.autograd.gradcheck.GradcheckError: Backward is not reentrant, i.e., running backward with same input and grad_output multiple times gives different values, although analytical gradient matches numerical gradient.The tolerance for nondeterminism was 0.0.

NOTE: If your op relies on non-deterministic operations i.e., it is listed here:
https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html
this failure might be expected.

If you are adding a new operator, please file an issue and then use one of the
workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck.
If the test
- manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck
  with `nondet_tol=<tol>` as a keyword argument.
- is OpInfo-based (e.g., in test_ops.py), then modify the OpInfo for the test
  to have `gradcheck_nondet_tol=<tol>`.
- is a Module test (e.g., in common_nn.py), then modify the corresponding
  module_test entry to have `gradcheck_nondet_tol=<tol>`
```

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @mruberry @kurtamohler