Repository: pytorch
Issue: test_foreach failing cuda memory leak check
Labels: module: cuda,triaged,module: mta

### 🐛 Describe the bug

Tests in test_foreach are failing the cuda memory leak check.  

Examples:
```
2023-02-21T09:30:46.6234306Z ======================================================================
2023-02-21T09:30:46.6234581Z ERROR [1.728s]: test_binary_op__foreach_pow_is_fastpath_False_cuda_float64 (__main__.TestForeachCUDA)
2023-02-21T09:30:46.6234852Z ----------------------------------------------------------------------
2023-02-21T09:30:46.6234986Z Traceback (most recent call last):
2023-02-21T09:30:46.6235370Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2069, in wrapper
2023-02-21T09:30:46.6235490Z     method(*args, **kwargs)
2023-02-21T09:30:46.6235862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2068, in wrapper
2023-02-21T09:30:46.6235972Z     with policy():
2023-02-21T09:30:46.6236346Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1608, in __exit__
2023-02-21T09:30:46.6236471Z     raise RuntimeError(msg)
2023-02-21T09:30:46.6236913Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestForeachCUDA.test_binary_op__foreach_pow_is_fastpath_False_cuda_float64! Caching allocator allocated memory was 4407296 and is now reported as 5606912 on device 0. CUDA driver allocated memory was 454033408 and is now 456130560.
2023-02-21T09:30:46.6236948Z 
2023-02-21T09:30:46.6237074Z ======================================================================
2023-02-21T09:30:46.6237307Z ERROR [1.419s]: test_binary_op__foreach_pow_is_fastpath_True_cuda_float32 (__main__.TestForeachCUDA)
2023-02-21T09:30:46.6237574Z ----------------------------------------------------------------------
2023-02-21T09:30:46.6237708Z Traceback (most recent call last):
2023-02-21T09:30:46.6238126Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2069, in wrapper
2023-02-21T09:30:46.6238244Z     method(*args, **kwargs)
2023-02-21T09:30:46.6238602Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2068, in wrapper
2023-02-21T09:30:46.6238715Z     with policy():
2023-02-21T09:30:46.6239069Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1608, in __exit__
2023-02-21T09:30:46.6239194Z     raise RuntimeError(msg)
2023-02-21T09:30:46.6239650Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestForeachCUDA.test_binary_op__foreach_pow_is_fastpath_True_cuda_float32! Caching allocator allocated memory was 7473152 and is now reported as 8095232 on device 0. CUDA driver allocated memory was 462422016 and is now 464519168.
2023-02-21T09:30:46.6239672Z 
2023-02-21T09:30:46.6239808Z ======================================================================
2023-02-21T09:30:46.6240044Z ERROR [1.346s]: test_binary_op__foreach_pow_is_fastpath_True_cuda_float64 (__main__.TestForeachCUDA)
2023-02-21T09:30:46.6240306Z ----------------------------------------------------------------------
2023-02-21T09:30:46.6240442Z Traceback (most recent call last):
2023-02-21T09:30:46.6240820Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2069, in wrapper
2023-02-21T09:30:46.6240936Z     method(*args, **kwargs)
2023-02-21T09:30:46.6241332Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2068, in wrapper
2023-02-21T09:30:46.6241447Z     with policy():
2023-02-21T09:30:46.6241819Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1608, in __exit__
2023-02-21T09:30:46.6241941Z     raise RuntimeError(msg)
2023-02-21T09:30:46.6242398Z RuntimeError: CUDA driver API confirmed a leak in __main__.TestForeachCUDA.test_binary_op__foreach_pow_is_fastpath_True_cuda_float64! Caching allocator allocated memory was 10760192 and is now reported as 11648512 on device 0. CUDA driver allocated memory was 470810624 and is now 472907776.
2023-02-21T09:30:46.6242418Z 
2023-02-21T09:30:46.6242683Z ----------------------------------------------------------------------
2023-02-21T09:30:46.6242845Z Ran 1166 tests in 542.496s
2023-02-21T09:30:46.6242864Z 
2023-02-21T09:30:46.6243001Z FAILED (errors=3, expected failures=9)
2023-02-21T09:30:46.6243021Z 
2023-02-21T09:30:46.6243145Z Generating XML reports...
2023-02-21T09:30:46.6243511Z Generated XML report: test-reports/python-unittest/test_foreach/TEST-TestForeachCUDA-20230221092140.xml
2023-02-21T09:30:46.6243883Z FINISHED PRINTING LOG FILE of test_foreach (/var/lib/jenkins/workspace/test/test-reports/test_foreach_81jyujhz.log)
2023-02-21T09:30:46.6243903Z 
2023-02-21T09:30:46.6244016Z test_foreach failed!
2023-02-21T09:30:46.6244148Z Traceback (most recent call last):
2023-02-21T09:30:46.6244342Z   File "/var/lib/jenkins/workspace/test/run_test.py", line 1394, in <module>
2023-02-21T09:30:46.6244439Z     main()
2023-02-21T09:30:46.6244627Z   File "/var/lib/jenkins/workspace/test/run_test.py", line 1352, in main
2023-02-21T09:30:46.6244746Z     raise RuntimeError(
2023-02-21T09:30:46.6244869Z RuntimeError: test_foreach failed!
2023-02-21T09:30:46.6244888Z 
```

https://hud.pytorch.org/pytorch/pytorch/commit/1ab112cfab5e9e5b3ec2521f0b4e6b93b6ff90d9
https://github.com/pytorch/pytorch/actions/runs/4230722115/jobs/7348826802
https://github.com/pytorch/pytorch/actions/runs/4230706718/jobs/7348783980
https://github.com/pytorch/pytorch/actions/runs/4230706718/jobs/7348788323
https://github.com/pytorch/pytorch/actions/runs/4230706718/jobs/7348957399
https://github.com/pytorch/pytorch/actions/runs/4230720847/jobs/7348827635

Last known success: 4d753b50451607b3314f827993df7e5527f0c0a7
First failure: 1ab112cfab5e9e5b3ec2521f0b4e6b93b6ff90d9

### Versions

CI

cc @crcrpar @mcarilli @ngimel