Repository: pytorch
Issue: [dataloader] RuntimeError: Too many open files when yielding integers
Labels: module: dataloader,triaged

## 🐛 Bug

I have a custom IterableDataset that reads words from a file and yields their unique ids, and whenever I set num_workers > 0 while yielding the word id, I get a RuntimeError "Too many open files. Communication with the workers is no longer possible.". On the other hand, if I'm yielding the word string, nothing happens.

## To Reproduce

Steps to reproduce the behavior:

IterableDataset code:
```python
import os
import torch
import tqdm
from torch.utils.data import IterableDataset, DataLoader

class MyIterableDataset(IterableDataset):
    def __init__(self, file_path, num_workers=4):
        super(MyIterableDataset).__init__()
        self.file_path = file_path
        self.file_size = os.stat(file_path).st_size

        chunk_size = self.file_size // num_workers
        start = 0
        end = chunk_size
        bonus = self.file_size - chunk_size * num_workers
        if (bonus > 0):
            end = chunk_size + 1
            bonus -= 1
        self.endpoints = [(start, end)]
        for i in range(1, num_workers):
            start = end
            if (bonus > 0):
                end += chunk_size + 1
                bonus -= 1
            else:
                end += chunk_size
            self.endpoints.append((start, end))
        print(self.endpoints)

    def read_word(self, f):
        ch = ''
        word = ""
        while True:
            ch = f.read(1)
            if not ch:
                return ''
            if (str.isspace(ch)):
                if len(word) > 0:
                    break
                if (ch == '\n'):
                    return "\n"
                else:
                    continue
            word += ch
        return word

    def parse_file(self, f, start, words_to_read, id):
        words_read = 0
        f.seek(start, 0)
        if id > 0:
            while True:
                ch = f.read(1)
                if not ch or str.isspace(ch):
                    break
                start += 1
            f.seek(start, 0)
        while True:                
            word = self.read_word(f)
            if word and word != "\n":
                words_read += 1
                yield 1  # I return 1 since the function to get the word id comes from a C++ class bound to python with pybind11
            if f.tell() >= words_to_read or not word:
                break
        r.close()
                
    def __iter__(self):
        f = open(self.file_path, "r")
        worker_info = torch.utils.data.get_worker_info()
        worker_id = 0 if worker_info is None else worker_info.id
        start, end = self.endpoints[worker_id]
        return self.parse_file(f, start, end, worker_id)

file_path = "./text8"
num_workers = 5
ds = MyIterableDataset(file_path=file_path, num_workers=num_workers)
tokens = [j for j in tqdm.tqdm(DataLoader(ds, num_workers=num_workers-1, batch_size=10))]
```
As I specified in the code comment, the function that returns a word id comes from a C++ class bound to python with pybind11, so I don't have here the code, but it doesn't matter because if the error is raised even if one returns an integer. If one instead yields 'word', nothing happen, but with degraded performance wrt single worker.   
The error I get is:
```python
RuntimeError                              Traceback (most recent call last)
<ipython-input-30-8a1ba426e0c3> in <module>
      2 
      3 t0 = time.time()
----> 4 tokens = [j for j in tqdm.tqdm(DataLoader(ds, num_workers=num_workers-1, batch_size=10))]
      5 print()
      6 print(time.time() - t0)

<ipython-input-30-8a1ba426e0c3> in <listcomp>(.0)
      2 
      3 t0 = time.time()
----> 4 tokens = [j for j in tqdm.tqdm(DataLoader(ds, num_workers=num_workers-1, batch_size=10))]
      5 print()
      6 print(time.time() - t0)

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/tqdm/std.py in __iter__(self)
   1165 
   1166         try:
-> 1167             for obj in iterable:
   1168                 yield obj
   1169                 # Update and possibly print the progressbar.

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_data(self)
   1066 
   1067             assert not self._shutdown and self._tasks_outstanding > 0
-> 1068             idx, data = self._get_data()
   1069             self._tasks_outstanding -= 1
   1070             if self._dataset_kind == _DatasetKind.Iterable:

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _get_data(self)
   1032         else:
   1033             while True:
-> 1034                 success, data = self._try_get_data()
   1035                 if success:
   1036                     return data

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)
    897             except OSError as e:
    898                 if e.errno == errno.EMFILE:
--> 899                     raise RuntimeError(
    900                         "Too many open files. Communication with the"
    901                         " workers is no longer possible. Please increase the"

RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code
```

## Expected behavior

I expect that If increase the number of workers and I yield the word id no error will be raised

## Environment

```
Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Pop!_OS 20.10 (x86_64)
GCC version: (Ubuntu 10.2.0-13ubuntu1) 10.2.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.7.1               py3.8_cpu_0  [cpuonly]  pytorch
[conda] torchvision               0.8.2                  py38_cpu  [cpuonly]  pytorch
```

## Additional context

Get text8 data with `wget http://mattmahoney.net/dc/enwik8.zip`

cc @SsnL @VitalyFedyunin @ejguan