Repository: pytorch
Issue: Apparent Memory Leak with torch.as_tensor
Labels: module: memory usage,triaged,module: numpy,quansight-nack,module: tensor creation

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

I am facing a very strange memory leak when using `torch.as_tensor` or `torch.tensor` only in a specific situation. I have placed a complete repro below, which demonstrates the issue clearly and reliably. The repro is reconstructed from an RL environment which accounts for the structure of the various objects and control flow.

Essentially, depending on how the object passed to `torch.as_tensor()` is constructed, tensors will refuse to deallocate upon the exit of the scope. A simple change causes tensors to deallocate as expected. Refer to the repro code for a complete example and an example of the change which fixes the allocation issue.

## To Reproduce

Steps to reproduce the behavior:

1. Run the below code. Across multiple runs, change the value of `repro_tensor_allocations` in the main function and observe correlated increases/decreases in memory usage after the buffer is completely filled. The tensors created in proportion with  `repro_tensor_allocations` do not deallocate.

2. In the function `fill_buf` replace the two `fake_data_batches` assignment lines with the commented out `fake_data_batches` line. Observe that memory usage is no longer correlated with the value of `repro_tensor_allocations`. The tensors created in proportion with  `repro_tensor_allocations` do now deallocate.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

```python
from collections import deque

import numpy as np
import time
import torch, random


class FrameHolder:
    def __init__(self, frames):
        self.frames = frames

    def get(self):
        return np.concatenate(self.frames, axis=0)


def convert(data):
    data_t = torch.as_tensor(data, dtype=torch.float32)

    return random.randint(1, 100)


def fill_buf(data_buf, n, repro_tensor_allocation):
    for i in range(n):
        fake_data = np.random.uniform(size=(4, 4, 50, 50))
        fake_data_frames = [FrameHolder([np.expand_dims(f, axis=0) for f in fake_data[b, :, :, :]]) for b in
                            range(len(fake_data))]

        fake_data_batches = [f.get() for f in fake_data_frames]
        fake_data_batches = np.stack(fake_data_batches)

        # Replace above two lines with this line to fix memory leak
        # fake_data_batches = np.stack([f.get() for f in fake_data_frames])

        for _ in range(repro_tensor_allocation):
            fake_action = convert(fake_data_batches)

        data_buf.append(dict(a=fake_data_frames, b=fake_action))

        if i % 1000 == 0:
            print(i, n)

    return


def main():
    n = int(5e3)
    repro_tensor_allocations = 5
    data_buf = deque(maxlen=n)
    fill_buf(data_buf, n, repro_tensor_allocations)

    print('filled')

    while True:
        time.sleep(1)


if __name__ == '__main__':
    main()
```

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
In the repro I demonstrate a fix which exhibits the expected behavior. Namely, tensors deallocating as expected.

## Environment

 - PyTorch Version (e.g., 1.0): 1.7.1
 - OS (e.g., Linux): Ubuntu 18.04
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source): N/A
 - Python version: 3.6.9
 - CUDA/cuDNN version: 10.2
 - GPU models and configuration: 1080ti, single
 - Any other relevant information: Also observed in docker ubuntu:18.04, and in python 3.8.1, and in pytorch 1.7.0

## Additional context

<!-- Add any other context about the problem here. -->


cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @mruberry @rgommers @heitorschueroff