Repository: pytorch
Issue: DISABLED test_prod_large (__main__.TestCuda)
Labels: module: cuda,triaged,module: flaky-tests,skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_prod_large&suite=TestCuda&limit=100) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/29382477213).

Over the past 3 hours, it has been determined flaky in 5 workflow(s) with 15 failures and 5 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_prod_large`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.



<details><summary>Sample error message</summary>

```
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1159, in originate_pairs
    return [pair_type(actual, expected, id=id, **options)]
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2610, in __init__
    super().__init__(actual, expected, check_dtype=False, **other_parameters)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 536, in __init__
    actual, expected = self._process_inputs(actual, expected, id=id)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2625, in _process_inputs
    return [self._to_number(input, id=id) for input in (actual, expected)]
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2625, in <listcomp>
    return [self._to_number(input, id=id) for input in (actual, expected)]
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2637, in _to_number
    number = number_like.item()
RuntimeError: Host and device pointer dont match with cudaHostRegister. Please dont use this feature by setting PYTORCH_CUDA_ALLOC_CONF=use_cuda_host_register:False (default)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/test_cuda.py", line 816, in test_prod_large
    self.assertEqual(x.prod(), 1)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3846, in assertEqual
    error_metas = not_close_error_metas(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1217, in not_close_error_metas
    pairs = originate_pairs(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1172, in originate_pairs
    f"Originating a {pair_type.__name__}() at item {''.join(str([item]) for item in id)} with\n\n"
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 993, in __format__
    return self.item().__format__(format_spec)
RuntimeError: Host and device pointer dont match with cudaHostRegister. Please dont use this feature by setting PYTORCH_CUDA_ALLOC_CONF=use_cuda_host_register:False (default)

To execute this test, run the following from the base repo dir:
    python test/test_cuda.py TestCuda.test_prod_large

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```

</details>


Test file path: `test_cuda.py`

cc @ptrblck @msaroufim @clee2000