Repository: pytorch
Issue: DISABLED test_fn_grad_grid_sampler_2d_cuda_float64 (__main__.TestBwdGradientsCUDA)
Labels: triaged,module: flaky-tests,skipped,module: unknown

Platforms: inductor, rocm



This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_fn_grad_grid_sampler_2d_cuda_float64&suite=TestBwdGradientsCUDA&limit=100) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/27629805966).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_fn_grad_grid_sampler_2d_cuda_float64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.



<details><summary>Sample error message</summary>

```
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 1127, in test_wrapper
    return test(*args, **kwargs)
  File "test_ops_gradients.py", line 37, in test_fn_grad
    self._grad_test_helper(device, dtype, op, op.get_op())
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 5044, in _grad_test_helper
    return self._check_helper(device, dtype, op, variant, 'gradcheck', check_forward_ad=check_forward_ad,
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 5013, in _check_helper
    self.assertTrue(gradcheck(fn, gradcheck_args,
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 4583, in gradcheck
    return torch.autograd.gradcheck(fn, inputs, **kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 2053, in gradcheck
    return _gradcheck_helper(**args)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 2082, in _gradcheck_helper
    _gradcheck_real_imag(
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 1492, in _gradcheck_real_imag
    gradcheck_fn(
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 1922, in _fast_gradcheck
    analytical_vJu = _get_analytical_vJu_backward_mode(
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 805, in _get_analytical_vJu_backward_mode
    all_vJ = _check_analytical_jacobian_attributes(
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/autograd/gradcheck.py", line 791, in _check_analytical_jacobian_attributes
    raise GradcheckError(
torch.autograd.gradcheck.GradcheckError: Backward is not reentrant, i.e., running backward with same input and grad_output multiple times gives different values, although analytical gradient matches numerical gradient.The tolerance for nondeterminism was 1e-15.

NOTE: If your op relies on non-deterministic operations i.e., it is listed here:
https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html
this failure might be expected.

If you are adding a new operator, please file an issue and then use one of the
workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck.
If the test
- manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck
  with `nondet_tol=<tol>` as a keyword argument.
- is OpInfo-based (e.g., in test_ops_gradients.py), then modify the OpInfo for the test
  to have `gradcheck_nondet_tol=<tol>`.
- is a Module test (e.g., in common_nn.py), then modify the corresponding
  module_test entry to have `gradcheck_nondet_tol=<tol>`


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2829, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2829, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 447, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 1139, in test_wrapper
    raise e_tracked from e
Exception: Caused by sample input at index 14: SampleInput(input=Tensor[size=(2, 3, 5, 20), device="cuda:0", dtype=torch.float64], args=(Tensor[size=(2, 13, 10, 2), device="cuda:0", dtype=torch.float64],2,1,False), kwargs={}, broadcasts_input=False, name='')

To execute this test, run the following from the base repo dir:
    PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=14 PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_ops_gradients.py -k TestBwdGradientsCUDA.test_fn_grad_grid_sampler_2d_cuda_float64

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```

</details>


Test file path: `test_ops_gradients.py`

cc @clee2000 @ezyang @anijain2305 @chauhang @penguinwu