Repository: pytorch
Issue: LSTMs leak memory in CPU PyTorch 1.5.1, 1.6, and 1.7 on Linux
Labels: high priority,module: rnn,module: cpu,module: memory usage,triaged

## 🐛 Bug

On Linux, when using the CPU (whether a CPU device in a CUDA-capable PyTorch, or using a CPU-only PyTorch distribution), merely _instantiating_ modules with an LSTM in them claims memory that is never released until the process is killed. _Training_ using one of these modules consumes more and more memory during training, in what seems to be a nondeterministic way. I have reproduced this in PyTorch 1.5.1, 1.6.0, and 1.7.0 on Linux.

Oddly, the _MacOS_ CPU-only PyTorch distribution _does not_ leak in this way.

I was not able to resolve the leak in the below allocation test (tested with the 1.6 nightly) using any combination of `MKL_DISABLE_FAST_MM=1` or `OMP_NUM_THREADS=4`, so I believe this leak is distinct from other issues I came across trying to troubleshoot it.

## To Reproduce

Steps to reproduce the behavior:

1. On Linux, using any of PyTorch `1.5.1+cpu`, `1.6.0.dev20200625+cpu`, or `1.7.0.dev20200715+cpu`, run the below allocation_test.py script
1. Using `htop`, find the PID output near the top of the script. You should see that the resident set (`RES`) for this process in memory is near 1GB (on my machine, 946M).
1. Once it finishes the deallocation step, note that a very small amount of memory has been recovered, but most of it has not been.
1. Repeat these steps on MacOS and notice that the memory is almost entirely recovered (on my machine, a little under 100MB is left resident, and increasing the size of the test doesn't increase this much on MacOS).

```{python}
# allocation_test.py
import os
import torch
import torch.nn as nn
import gc
from time import sleep

print(os.getpid())
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
throwaway = torch.ones((1,1)).to(device) # load CUDA context

class Encoder(nn.Module):
    def __init__(self, input_dim, hidden_dim, n_layers, dropout_perc):
        super().__init__()
        self.hidden_dim, self.n_layers = (hidden_dim, n_layers)
        self.rnn = nn.LSTM(input_dim,hidden_dim,n_layers,dropout=dropout_perc)
    def forward(self,x):
        outputs, (hidden, cell) = self.rnn(x)
        return hidden, cell

print('allocating memory')
pile=[]
for i in range(1500):
    pile.append(Encoder(102,64,4,0.5).to(device))

print('waiting two seconds')
sleep(2)

print('hypothetically de-allocating memory')
del pile
gc.collect()
if torch.cuda.is_available():
    torch.cuda.empty_cache()

print('waiting forever')
while True:
    sleep(1)
```

## Expected behavior

The MacOS behavior (near-immediate full recovery of all memory allocated by the LSTM module) is expected on Linux as well.

## Environment

Collecting environment information...
PyTorch version: 1.5.1+cpu
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.8
Is CUDA available: No
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 440.100
cuDNN version: /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn.so.7.2.1

Versions of relevant libraries:
[pip3] numpy==1.19.0
[pip3] torch==1.5.1+cpu
[conda] Could not collect

## Additional context

There is some other diagnostic information and discussion here:
https://discuss.pytorch.org/t/lstm-on-cpu-wont-release-memory-when-all-refs-deleted/89026


cc @ezyang @gchanan @zou3519 @VitalyFedyunin