Repository: pytorch
Issue: Memory Leak Found in Persistent DataLoader
Labels: module: dataloader,module: memory usage,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior: Run the provided code snippet
```python
import os

import numpy as np
import psutil
from torch.utils.data import DataLoader, Dataset


class MyDataset(Dataset):
    def __init__(self):
        self.data = np.random.rand(10000, 224, 244, 3)

    def __len__(self):
        return self.data.shape[0]

    def __getitem__(self, idx):
        return self.data[idx] + 1.0

if __name__ == "__main__":
    loader = DataLoader(MyDataset(), batch_size=4096, num_workers=os.cpu_count(), persistent_workers=True)
    for epoch in range(9):
        for step, _ in enumerate(loader):
            print("Epoch: {}, step: {}, RAM used: {:.2f} GB".format(epoch, step, psutil.virtual_memory().used / 1e9))
```

## Expected behavior
When running the code snippet as is, you will see the RAM usage keeps getting increased. (you can also open `htop` to monitor RAM usage as well)
```
Epoch: 0, step: 0, RAM used: 14.45 GB
Epoch: 0, step: 1, RAM used: 14.44 GB
Epoch: 0, step: 2, RAM used: 14.44 GB
Epoch: 1, step: 0, RAM used: 14.46 GB
Epoch: 1, step: 1, RAM used: 14.46 GB
Epoch: 1, step: 2, RAM used: 14.46 GB
Epoch: 2, step: 0, RAM used: 14.47 GB
Epoch: 2, step: 1, RAM used: 14.47 GB
Epoch: 2, step: 2, RAM used: 14.47 GB
Epoch: 3, step: 0, RAM used: 25.24 GB
Epoch: 3, step: 1, RAM used: 25.24 GB
Epoch: 3, step: 2, RAM used: 25.24 GB
Epoch: 4, step: 0, RAM used: 36.00 GB
Epoch: 4, step: 1, RAM used: 36.00 GB
Epoch: 4, step: 2, RAM used: 36.00 GB
Epoch: 5, step: 0, RAM used: 41.39 GB
Epoch: 5, step: 1, RAM used: 41.39 GB
Epoch: 5, step: 2, RAM used: 41.39 GB
Epoch: 6, step: 0, RAM used: 41.41 GB
Epoch: 6, step: 1, RAM used: 41.40 GB
Epoch: 6, step: 2, RAM used: 41.40 GB
Epoch: 7, step: 0, RAM used: 41.41 GB
Epoch: 7, step: 1, RAM used: 41.41 GB
Epoch: 7, step: 2, RAM used: 41.41 GB
Epoch: 8, step: 0, RAM used: 46.80 GB
Epoch: 8, step: 1, RAM used: 46.79 GB
Epoch: 8, step: 2, RAM used: 46.79 GB
```
In comparison, when setting the `persistent_worker=False`, the memory is stable and the print looks like: 
```
Epoch: 0, step: 0, RAM used: 14.46 GB
Epoch: 0, step: 1, RAM used: 14.47 GB
Epoch: 0, step: 2, RAM used: 14.47 GB
Epoch: 1, step: 0, RAM used: 14.48 GB
Epoch: 1, step: 1, RAM used: 14.47 GB
Epoch: 1, step: 2, RAM used: 14.47 GB
Epoch: 2, step: 0, RAM used: 14.47 GB
Epoch: 2, step: 1, RAM used: 14.47 GB
Epoch: 2, step: 2, RAM used: 14.47 GB
Epoch: 3, step: 0, RAM used: 14.48 GB
Epoch: 3, step: 1, RAM used: 14.47 GB
Epoch: 3, step: 2, RAM used: 14.47 GB
Epoch: 4, step: 0, RAM used: 14.80 GB
Epoch: 4, step: 1, RAM used: 14.47 GB
Epoch: 4, step: 2, RAM used: 14.47 GB
Epoch: 5, step: 0, RAM used: 14.47 GB
Epoch: 5, step: 1, RAM used: 14.47 GB
Epoch: 5, step: 2, RAM used: 14.47 GB
Epoch: 6, step: 0, RAM used: 14.47 GB
Epoch: 6, step: 1, RAM used: 14.47 GB
Epoch: 6, step: 2, RAM used: 14.47 GB
Epoch: 7, step: 0, RAM used: 14.47 GB
Epoch: 7, step: 1, RAM used: 14.47 GB
Epoch: 7, step: 2, RAM used: 14.47 GB
Epoch: 8, step: 0, RAM used: 14.47 GB
Epoch: 8, step: 1, RAM used: 14.47 GB
Epoch: 8, step: 2, RAM used: 14.47 GB
```

## Environment
Pull official PyTorch docker image by `pytorch/pytorch:1.9.0-cuda10.2-cudnn7-runtime`.  Then create a container and run the code above inside container.
```

 - PyTorch Version : 1.9.0 (provided by container)
 - OS: Host is Linux, the container is Linux too.
 - How you installed PyTorch: PyTorch is pre-installed in container
 - Python version: 3.7.10 (provided by docker image)
 - CUDA/cuDNN version: 10.2 (provided by docker image)
 - GPU models and configuration: Nvidia V100 (but given the code snippet has no GPU-related code, it shouldn't matter)
```
cc @SsnL @VitalyFedyunin @ejguan