Repository: pytorch
Issue: Inference ran on new thread leak memory on Android
Labels: module: memory usage,module: android,oncall: mobile

## 🐛 Bug

When running a model inference in a new thread, the memory consumption of the application keeps going up at each inference.
When running the same model from the main thread maintains a steady memory consumption.

This only seems to appear on Android, the same code ran on Linux does not seem to have that issue

Of course, after a certain amount of inferences, the OOM killer might start being unhappy and killing apps on the device.

## To Reproduce

Steps to reproduce the behavior:

1. Load the model
2. Create threads running inference (even sequentially created threads)
3. Monitor memory consumption with top or other and witness free memory reducing

Model to reproduce:

```python
#! /usr/bin/python3                                                                                                                                                                                                                    

import torch

class ThreadMe(torch.nn.Module):
    def __init__(self):
        super(ThreadMe, self).__init__()
        self.conv2d = torch.nn.Conv2d(3, 3, 3)

    def forward(self, x):
        return self.conv2d(x)


model = ThreadMe()
model.eval()
trace_input = torch.rand(1, 3, 600, 800)
with torch.no_grad():
    script_model = torch.jit.trace(model, trace_input)
torch.jit.save(script_model, "reproducer.pt")
```

C++ loading code:

```C++
#include <torch/script.h>
#include <iostream>
#include <future>
#include <exception>
#include <unistd.h>

void run_inference(torch::jit::script::Module& model, torch::Tensor& a, torch::Tensor& out)
{
    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(a);

    out = model.forward(inputs).toTensor();
}

int main(int argc, char*argv[])
{
    auto model = torch::jit::load("reproducer.pt", torch::Device(torch::kCPU));
    constexpr unsigned int ninferences = 200;
    std::cout << "Model loaded" << std::endl;
    sleep(10);

    for (uint32_t i = 0; i < ninferences; ++i) {
        auto tensor_a = at::rand({1, 3, 600, 800}, at::device(at::kCPU).dtype(at::kFloat));
        torch::Tensor out;
        std::future<void> f;

        f = std::async(std::launch::async,
                       run_inference,
                       std::ref(model),
		       std::ref(tensor_a),
		       std::ref(out));
        if (!f.valid())
             throw std::runtime_error("Something went wrong");
        f.wait();
    }
}
```

## Expected behavior

Small spikes of memory consumption at each inference but going back down after each inference.

## Environment
Run environment:
(Don't have python on the Android device)
- Android, arm64-v8a, 8 cores
- Libtorch: pytorch_android-1.8.0.aar release

Model export environment:
PyTorch version: 1.8.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.10.2

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce MX150
Nvidia driver version: 465.19.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] pytorch-msssim==0.2.1
[pip3] torch==1.8.1
[pip3] torchvision==0.9.1
[conda] Could not collect