Problems with MTEncDecModel model not releasing GPU memory
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I am creating an MTEncDecModel model and the gpu memory allocated never gets freed. 

Specifically, I want to create an MTEncDecModel model for english to german, perform some translations, delete then model, then create an MTEncDecModel model for english to french, perform some translations, delete the model, and continue this process for many language pairs. However, the gpu memory allocated to each model instantiated is not freed, so I get a `RuntimeError: CUDA Out of memory` error.

I want a way to clear the gpu memory allocated without terminating the python program or restarting the jupyter kernel.

**Steps/Code to reproduce bug**

```
import torch
from nemo.collections.nlp.models import MTEncDecModel
from transformers import MarianMTModel

def memory_stats():
    print('Memory allocated:', torch.cuda.memory_allocated()/1024**2)
    print('Memory cached:', torch.cuda.memory_cached()/1024**2, '\n')

print('Initia memory stats')
memory_stats()

model = MTEncDecModel.from_pretrained('nmt_en_es_transformer24x6').cuda()
print('Model created')
memory_stats()

del model
print('Model deleted')
memory_stats()

torch.cuda.empty_cache()
print('Cache emptied')
memory_stats()
```

The output from this code is
```
Initial memory stats
Memory allocated: 0.0
Memory cached: 0.0 

Model created
Memory allocated: 1793.8173828125
Memory cached: 3714.0 

Model deleted
Memory allocated: 1793.8173828125
Memory cached: 3714.0 

Cache emptied
Memory allocated: 1793.8173828125
Memory cached: 1802.0 
```

**Expected behavior**
Deleting all objects and references pointing to objects allocating GPU memory should free the memory. Link: 
https://discuss.pytorch.org/t/free-all-gpu-memory-used-in-between-runs/168202/2

For example, If I create a PyTorch tensor instead of the MTEncDecModel, then the output is as expected:
```
print('Initial memory stats')
memory_stats()

x = torch.randn(1024*1024, device='cuda')
print('Tensor created')
memory_stats()

del x
print('Tensor deleted')
memory_stats()

torch.cuda.empty_cache()
print('Cache emptied')
memory_stats()
```

Output:
```
Initial memory stats
Memory allocated: 0.0
Memory cached: 0.0 

Tensor created
Memory allocated: 4.0
Memory cached: 20.0 

Tensor deleted
Memory allocated: 0.0
Memory cached: 20.0 

Cache emptied
Memory allocated: 0.0
Memory cached: 0.0 
```

Also, if I create a Marian machine translation model from HuggingFace instead of the MTEncDecModel model then again the output is as expected:
```
print('Initial memory stats')
memory_stats()

model = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-tc-big-en-it').cuda()
print('Model created')
memory_stats()

del model
print('Model deleted')
memory_stats()

torch.cuda.empty_cache()
print('Cache emptied')
memory_stats()
```

Output
```
Initial memory stats
Memory allocated: 0.0
Memory cached: 0.0 

Model created
Memory allocated: 893.5439453125
Memory cached: 896.0 

Model deleted
Memory allocated: 0.0
Memory cached: 896.0 

Cache emptied
Memory allocated: 0.0
Memory cached: 0.0 
```

In fact, even instantiating an MTEncDecModel without sending to model to `cuda()` and without storing the model to a variable, causes the memory allocated to increate every time:
```
print('Initial memory stats')
memory_stats()

MTEncDecModel.from_pretrained('nmt_en_es_transformer24x6')
torch.cuda.empty_cache()
print('After one instantiation')
memory_stats()

MTEncDecModel.from_pretrained('nmt_en_es_transformer24x6')
torch.cuda.empty_cache()
print('After two instantiations')
memory_stats()

MTEncDecModel.from_pretrained('nmt_en_es_transformer24x6')
torch.cuda.empty_cache()
print('After three instantiations')
memory_stats()
```

Output:
```
Initial memory stats
Memory allocated: 0.0
Memory cached: 0.0

After one instantiation
Memory allocated: 1793.8173828125
Memory cached: 1802.0

After two instantiations
Memory allocated: 3587.634765625
Memory cached: 3588.0 

After three instantiations
Memory allocated: 5381.4521484375
Memory cached: 5390.0 
```

**Environment overview (please complete the following information)**

 - Environment location: Docker on Kubernetes
 - Method of NeMo install: using helm chart using nvcr.io/nvidia/nemo: 1.5.1