Repository: pytorch
Issue: Multi-processing leaking file descriptors
Labels: module: multiprocessing,module: dataloader,triaged

## 🐛 Bug
Multiprocessing leaks file descriptors unexpectedly. The issues #11201 and #973 leads us to some workarounds such as to set sharing to file system instead of descriptors, but I think this does not hit the core of the problem and it is also counter-intuitive.

## To Reproduce

Steps to reproduce the behavior:

1. Run the code (scenario set to 1 or 3).
2. Preferably, run `watch cat /proc/sys/fs/file-nr` in another terminal to watch the number of open file descriptors.

```python
import torch
from torch.utils.data import Dataset, DataLoader
from torch import multiprocessing
# the results are the same using standard MP, which makes me feel strange
# since the error is still raised in pytorch MP
import tqdm
import copy


def process_data(src):
    # Some mock processing logic
    return torch.rand([100])


class TestDataset(Dataset):
    def __len__(self):
        return 200000

    def __getitem__(self, index):
        return torch.rand([100])


if __name__ == "__main__":
    SCENARIO = 3
    if SCENARIO == 1:
        with multiprocessing.Pool(32) as pool:
            data_source = range(200000)  # mock source
            processed_data = list(tqdm.tqdm(pool.imap(process_data, data_source)))
            # Breaks at 65020 iterations here.
            # watch cat /proc/sys/fs/file-nr shows ever-increasing number of FD's
            torch.save(processed_data, 'test_data.pt')
    if SCENARIO == 2:
        with multiprocessing.Pool(32) as pool:
            data_source = range(200000)  # mock source
            processed_data = []
            for d in tqdm.tqdm(pool.imap(process_data, data_source)):
                processed_data.append(copy.deepcopy(d))
                del d
            # This runs successfully
            # watch cat /proc/sys/fs/file-nr shows steady number of FD's
            torch.save(processed_data, 'test_data.pt')
    if SCENARIO == 3:
        # Training model, would like to save data generated for reproducibility
        train_loader = DataLoader(TestDataset(), shuffle=True, num_workers=32)
        data_save = []
        for batch in tqdm.tqdm(train_loader):
            # do something with models
            data_save.append(batch)
        torch.save(data_save, 'test_data.pt')
```

The code produces the error below on the test server:
```
Traceback (most recent call last):
  File "test3.py", line 46, in <module>
    for batch in tqdm.tqdm(train_loader):
  File "/home/***/.local/lib/python3.6/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/***/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/***/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
    idx, data = self._get_data()
  File "/home/***/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
    success, data = self._try_get_data()
  File "/home/***/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/***/.local/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
    storage = cls._new_shared_fd(fd, size)
RuntimeError: unable to mmap 400 bytes from file <filename not specified>: Cannot allocate memory (12)
```

## Expected behavior
Scenario 1 and 3 should work without leaking file descriptors as in scenario 2: if there is a way to gain performance while fixing the leaking FD's then it is excellent; if it is related with performance of sharing tensors, it seems that the copy operation in scenario 2 does not incur much performance overhead, and in real world applications the dataset preprocessing will do heavier work so the overhead will be even smaller compared to the whole process.

If these are not possible, perhaps a warning should be added about possibly leaking FD, or the error message could tell a bit more about the situation. Having a memory allocation error with 750GB RAM available on the server is confusing to me. Wandering randomly on the internet, it takes me days to find the problem.

In addition, on the test server, setting sharing behavior to filesystem in the main block does not fix this problem.

## Environment
 - PyTorch Version (e.g., 1.0): 1.9.0
 - OS (e.g., Linux): Linux
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source): N/A
 - Python version: 3.6.9
 - CUDA/cuDNN version: 10.2
 - GPU models and configuration: P100
 - Any other relevant information:

## Additional context
People may generally use `numpy` for preprocessing. I'm using the `DGL` library with torch backend, so data processing is centered around torch instead.


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @gcramer23 @SsnL @VitalyFedyunin @ejguan @cbalioglu