Repository: pytorch
Issue: [CI] Multiple tests fail with `Host and device pointer dont match with cudaHostRegister` and are being disabled
Labels: high priority,module: cuda,module: ci,triaged

### 🐛 Describe the bug

Example failure is e.g. visible in: https://github.com/pytorch/pytorch/issues/136537
which disables `test_graph_optims_Adam_cuda_float32`.
Checking the logs however shows that this error might be a fallout of a previous error:
https://hud.pytorch.org/flakytest?name=test_graph_optims_Adam_cuda_float32&suite=TestCudaOptimsCUDA&limit=100:
```
2024-09-24T16:25:11.4725870Z test_cuda.py::TestCudaAutocast::test_cuda_autocast_deprecated_warning PASSED [0.0125s] [ 82%]
2024-09-24T16:25:11.4729838Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_AdamW_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135692 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 83%]
2024-09-24T16:25:11.4735935Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_Adam_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135721 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 83%]
2024-09-24T16:25:11.4742040Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_SGD_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135783 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 84%]
2024-09-24T16:25:11.4748098Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_AdamW_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135829 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 85%]
2024-09-24T16:25:11.4754146Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_Adam_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135881 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 85%]
2024-09-24T16:25:11.4760177Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_SGD_cuda_float32 SKIPPED [0.0012s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135925 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 86%]
2024-09-24T16:25:11.4764049Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_Adagrad_cuda_float32 SKIPPED [0.0020s] (cuda is not supported for fused on Adagrad) [ 86%]
2024-09-24T16:25:11.4768258Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_AdamW_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135963 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 87%]
2024-09-24T16:25:11.4774184Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_Adam_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/135993 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 87%]
2024-09-24T16:25:11.4780089Z test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_SGD_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136013 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 88%]
2024-09-24T16:25:11.4785951Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_AdamW_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136126 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 88%]
2024-09-24T16:25:11.4792299Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_Adam_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136137 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 89%]
2024-09-24T16:25:11.4798306Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_SGD_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136176 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 90%]
2024-09-24T16:25:11.4804303Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_AdamW_cuda_float32 SKIPPED [0.0012s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136230 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 90%]
2024-09-24T16:25:11.4810330Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_Adam_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136352 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 91%]
2024-09-24T16:25:11.4816211Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_SGD_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136372 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 91%]
2024-09-24T16:25:11.4822113Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_AdamW_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136411 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 92%]
2024-09-24T16:25:11.4828147Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_Adam_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136417 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 92%]
2024-09-24T16:25:11.4834142Z test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_SGD_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136437 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 93%]
2024-09-24T16:25:11.4839819Z test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_ASGD_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136468 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 93%]
2024-09-24T16:25:11.4845421Z test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adadelta_cuda_float32 SKIPPED [0.0009s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136506 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 94%]
2024-09-24T16:25:11.4850965Z test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_AdamW_cuda_float32 SKIPPED [0.0012s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/136521 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 95%]
2024-09-24T16:25:11.4854417Z test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adam_cuda_float32 ('RERUN', {'yellow': True}) [0.2080s] [ 95%]
2024-09-24T16:25:11.4856042Z test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adam_cuda_float32 ('RERUN', {'yellow': True}) [0.1982s] [ 95%]
2024-09-24T16:25:11.4857401Z test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adam_cuda_float32 FAILED [0.1977s] [ 95%]
2024-09-24T16:25:11.4858128Z 
2024-09-24T16:25:11.4858412Z ==================================== RERUNS ====================================
2024-09-24T16:25:11.4859282Z ____________ TestCudaOptimsCUDA.test_graph_optims_Adam_cuda_float32 ____________
2024-09-24T16:25:11.4860066Z Traceback (most recent call last):
2024-09-24T16:25:11.4861384Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1232, in not_close_error_metas
2024-09-24T16:25:11.4862541Z     pair.compare()
2024-09-24T16:25:11.4863613Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 711, in compare
2024-09-24T16:25:11.4864708Z     self._compare_values(actual, expected)
2024-09-24T16:25:11.4866018Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 841, in _compare_values
2024-09-24T16:25:11.4867090Z     compare_fn(
2024-09-24T16:25:11.4868328Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1023, in _compare_regular_values_close
2024-09-24T16:25:11.4869536Z     if torch.all(matches):
2024-09-24T16:25:11.4870914Z RuntimeError: Host and device pointer dont match with cudaHostRegister. Please dont use this feature by setting PYTORCH_CUDA_ALLOC_CONF=use_cuda_host_register:False (default)
```
shows multiple tests were already skipped before.
Visiting these shows the same error again; e.g. let's pick the first skipped one:
https://github.com/pytorch/pytorch/issues/135692

showing the same error again.

TL;DR: it seems the `cudaHostRegister` error creeps through CI and disables tests one-by-one blaming unrelated modules.

CC @crcrpar @janeyx99 @malfet 

### Versions

Visible in CI, so no specific versions. 

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @seemethere @malfet @pytorch/pytorch-dev-infra