Repository: pytorch
Issue: test_torch.py::TestTorchDeviceTypeCUDA::test_deepcopy_scalar_cuda_float32 and others leaking memory in inductor
Labels: triaged,oncall: pt2,module: inductor

Ex
https://github.com/pytorch/pytorch/actions/runs/10176833665/job/28149060529
https://github.com/pytorch/pytorch/actions/runs/10176833665/job/28149061493

```

2024-07-31T10:53:31.4545202Z __________ TestTorchDeviceTypeCUDA.test_bool_tensor_value_change_cuda __________
2024-07-31T10:53:31.4545308Z Traceback (most recent call last):
2024-07-31T10:53:31.4545704Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2918, in wrapper
2024-07-31T10:53:31.4545798Z     method(*args, **kwargs)
2024-07-31T10:53:31.4546190Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2918, in wrapper
2024-07-31T10:53:31.4546284Z     method(*args, **kwargs)
2024-07-31T10:53:31.4546673Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2917, in wrapper
2024-07-31T10:53:31.4546773Z     with policy():
2024-07-31T10:53:31.4547162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2348, in __exit__
2024-07-31T10:53:31.4547265Z     raise RuntimeError(msg)
2024-07-31T10:53:31.4548137Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestTorchDeviceTypeCUDA.test_bool_tensor_value_change_cuda! Caching allocator allocated memory was 2048 and is now reported as 3072 on device 0. CUDA driver allocated memory was 338034688 and is now 340131840.
2024-07-31T10:53:31.4548142Z 
2024-07-31T10:53:31.4548319Z To execute this test, run the following from the base repo dir:
2024-07-31T10:53:31.4548800Z     PYTORCH_TEST_WITH_INDUCTOR=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/test_torch.py -k TestTorchDeviceTypeCUDA.test_bool_tensor_value_change_cuda
2024-07-31T10:53:31.4548808Z 
2024-07-31T10:53:31.4549041Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-07-31T10:53:31.4549241Z ----------------------------- Captured stdout call -----------------------------
2024-07-31T10:53:31.4549338Z frames [('total', 2), ('ok', 2)]
2024-07-31T10:53:31.4549743Z inline_call [("raise <module 'torch._dynamo.exc' from '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py'>", 1)]
2024-07-31T10:53:31.4549841Z unimplemented []
2024-07-31T10:53:31.4550241Z graph_break [("raise <module 'torch._dynamo.exc' from '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py'>", 1)]
2024-07-31T10:53:31.4550380Z stats [('calls_captured', 4), ('unique_graphs', 1)]
2024-07-31T10:53:31.4550485Z inductor [('fxgraph_cache_miss', 1)]
2024-07-31T10:53:31.4550588Z aot_autograd [('total', 1), ('ok', 1)]
2024-07-31T10:53:31.4550774Z ----------------------------- Captured stdout call -----------------------------
2024-07-31T10:53:31.4550898Z frames [('total', 2), ('ok', 2)]
2024-07-31T10:53:31.4551419Z inline_call [("raise <module 'torch._dynamo.exc' from '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py'>", 1)]
2024-07-31T10:53:31.4551512Z unimplemented []
2024-07-31T10:53:31.4551907Z graph_break [("raise <module 'torch._dynamo.exc' from '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py'>", 1)]
2024-07-31T10:53:31.4552045Z stats [('calls_captured', 4), ('unique_graphs', 1)]
2024-07-31T10:53:31.4552152Z aot_autograd [('total', 1), ('ok', 1)]
2024-07-31T10:53:31.4552265Z inductor [('fxgraph_cache_miss', 1)]
2024-07-31T10:53:31.4552444Z ----------------------------- Captured stdout call -----------------------------
2024-07-31T10:53:31.4552540Z frames [('total', 2), ('ok', 2)]
2024-07-31T10:53:31.4552943Z inline_call [("raise <module 'torch._dynamo.exc' from '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py'>", 1)]
2024-07-31T10:53:31.4553035Z unimplemented []
2024-07-31T10:53:31.4553435Z graph_break [("raise <module 'torch._dynamo.exc' from '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py'>", 1)]
2024-07-31T10:53:31.4553578Z stats [('calls_captured', 4), ('unique_graphs', 1)]
2024-07-31T10:53:31.4553684Z aot_autograd [('total', 1), ('ok', 1)]
2024-07-31T10:53:31.4553800Z inductor [('fxgraph_cache_miss', 1)]
2024-07-31T10:53:31.4554237Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/test_torch/test_torch-4f55a794fda4d7f1.xml -
2024-07-31T10:53:31.4554477Z =========================== short test summary info ============================
2024-07-31T10:53:31.4555927Z FAILED [0.6891s] test_torch.py::TestTorchDeviceTypeCUDA::test_bool_tensor_value_change_cuda - RuntimeError: CUDA driver API confirmed a leak in __main__.TestTorchDeviceTypeCUDA.test_bool_tensor_value_change_cuda! Caching allocator allocated memory was 2048 and is now reported as 3072 on device 0. CUDA driver allocated memory was 338034688 and is now 340131840.
2024-07-31T10:53:31.4555938Z 
```

To enable the mem leak check on a PR, add the mem-leak-check label before the tests run

cc @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire