Repository: pytorch
Issue: Memory leak when moving tensors between Metal and CPU on iOS
Labels: oncall: mobile

## 🐛 Bug

Moving tensors between Metal and CPU causes memory leak i.e. memory still stays allocated even after the function call is finished successfully. 

## To Reproduce

I have an auto-encoder with latent manipulation linear algebra ops mini model. The workflow is encoder -> LA ops -> decoder. 

The encoder and decoder run on Metal whereas ops in latent manipulation model only support CPU. Hence, I have to move the output Metal tensors from encoder and decoder onto CPU. But doing this causes the memory leak. Any suggestions on how to fix this problem?

I have written two model inference functions for: 1) Metal and 2) CPU as follows in Objective-C++. They're pretty straightforward.

```objc
torch::Tensor inferCPUModel(NSString* modelPath, torch::Tensor inputTensor) {
    auto model = torch::jit::_load_for_mobile(modelPath.UTF8String);
    auto resultTensor = model.forward({inputTensor}).toTensor();
    return resultTensor;
}
```

```objc
torch::Tensor inferMetalModel(NSString* modelPath, torch::Tensor inputTensor) {
    auto model = torch::jit::_load_for_mobile(modelPath.UTF8String);
    auto resultTensor = model.forward({inputTensor.clone().metal()}).toTensor().cpu();
    return resultTensor;
}
```

The sequential tensor data flow through networks on Metal and CPU is as follows.

```objc
auto inputTensor = torch::randn({1, 3, 512, 512}, torch::kFloat);

// The flow of data is encoder (Metal) -> LAOps (CPU) -> decoder (Metal).
// All output tensors are on CPU.
auto outTensor1 = inferMetalModel(metalEncoderModelPath, inputTensor);
auto outTensor2 = inferCPUModel(cpuLinearAlgebraOpModelPath, outTensor1);
auto outTensor3 = inferMetalModel(metalDecoderModelPath, outTensor2);

// The flow is repeated multiple times, say 2.
auto outTensor4 = inferMetalModel(metalEncoderModelPath, inputTensor3);
auto outTensor5 = inferCPUModel(cpuLinearAlgebraOpModelPath, outTensor4);
auto outTensor6 = inferMetalModel(metalDecoderModelPath, outTensor5);

// The output is simply a pointer float buffer.
float *floatBuffer = outTensor6.data_ptr<float>();
```

At this point, the memory should be released because this giant sequential inference function was called from Swift. But, unfortunately, the memory is not released. When I create CPU models and all the inference is made on CPU then the memory does get released. 

I don't understand why moving tensors between Metal and CPU devices cause this memory leak. Is there any workaround or maybe my code is wrong? Any help is highly appreciated! 

## Expected behavior

Moving tensors between Metal and CPU should release the memory after the function call is completed. 

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 PyTorch version: 1.11.0.dev20211109
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.0.1 (x86_64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.3)
CMake version: version 3.22.0
Libc version: N/A

Python version: 3.9.9 (main, Nov 21 2021, 03:23:44)  [Clang 13.0.0 (clang-1300.0.29.3)] (64-bit runtime)
Python platform: macOS-12.0.1-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] pytorch-lightning==1.5.4
[pip3] pytorchvideo==0.1.3
[pip3] torch==1.11.0.dev20211109
[pip3] torchaudio==0.11.0.dev20211109
[pip3] torchmetrics==0.6.0
[pip3] torchtext==0.12.0.dev20211109
[pip3] torchvision==0.12.0.dev20211109
[conda] Could not collect

## Additional context

<!-- Add any other context about the problem here. -->