Repository: pytorch
Issue: Open file leak when dataloader is using persistent_workers and pin_memory AND you create multiple dataloaders.  
Labels: module: dataloader,triaged,module: data

### 🐛 Describe the bug

As the title says: if you create a dataloader multiple times, and the dataloader is using pin_memory and persistent_workers, then there is a file handle leak (OSError: [Errno 24] Too many open files). 

This is because of this line in torch.utils.data.Dataloader:
``` python
    if self._persistent_workers and self._pin_memory:
        import atexit
        for w in self._workers:
            atexit.register(_MultiProcessingDataLoaderIter._clean_up_worker, w)
```

In case (self._persistent_workers and self._pin_memory) is True, we are registering a clean up function. The atexit.register function opens up a file until the whole program exits. However, if we keep creating dataloaders, we keep opening files, hence the issue. 

Minimal code to reproduce:
``` python
import torch
from tqdm import tqdm
import numpy as np

class DummyDataSet:
    def __len__(self):
        return 5000

    def __getitem__(self, item):
        return np.zeros(10)


class Trainer:
    def __init__(self):
        self.trainloader = torch.utils.data.DataLoader(
            DummyDataSet(),
            batch_size=100,
            shuffle=True,
            drop_last=True,
            prefetch_factor=20,
            pin_memory=True,
            num_workers=20,
            persistent_workers=True)

    def train(self):
        for _ in range(100):
            for d in self.trainloader:
                pass


if __name__ == '__main__':
    for _ in tqdm(range(1000)):
        t = Trainer()
        t.train()
```
run, to show number of open files: 
```bash
ls /proc/proc_id/fd/ | wc -l
```
Temporary solution: don't create many dataloaders or don't use dataloaders with this specific arguments.

(Why would someone use a Trainer() in a for loop: the training is stochastic, this way we can collect multiple trainings and get a more accurate evaluation)



### Versions

PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.3.109
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080 Ti
Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.20.0
[pip3] torch==1.12.0
[pip3] torchelastic==0.2.0
[pip3] torchtext==0.13.0
[pip3] torchvision==0.13.0
[pip3] torchviz==0.0.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.20.0                   pypi_0    pypi
[conda] pytorch                   1.12.0          py3.7_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchelastic              0.2.0                    pypi_0    pypi
[conda] torchtext                 0.13.0                     py37    pytorch
[conda] torchvision               0.13.0               py37_cu113    pytorch
[conda] torchviz                  0.0.2                    pypi_0    pypi

cc @SsnL @VitalyFedyunin @ejguan @NivekT