Repository: pytorch
Issue: [dataloader] Hang because of too many open files (and probably some process dead)
Labels: module: dataloader,triaged

After many hours of training I got this. After this the training process is hung (maybe during running the finalizers). I imagine some workers are killed, and the main process somehow can't detect that.

Unfortunately the backtrace isn't very easy to parse (I forced a core dump using `kill -QUIT pid`)

```
Traceback (most recent call last):
  File "/miniconda/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
  File "/miniconda/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
  File "/miniconda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 327, in reduce_storage
  File "/miniconda/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
  File "/miniconda/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files
Traceback (most recent call last):
  File "/miniconda/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
  File "/miniconda/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
  File "/miniconda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 327, in reduce_storage
  File "/miniconda/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
  File "/miniconda/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files

root@faece257e5fe:/deepspeech.pytorch/convasr# Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
    finalizer()
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
    finalizer()
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/miniconda/lib/python3.7/shutil.py", line 491, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/miniconda/lib/python3.7/shutil.py", line 410, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/miniconda/lib/python3.7/shutil.py", line 406, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/miniconda/lib/python3.7/shutil.py", line 491, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
    finalizer()

OSError: [Errno 24] Too many open files: '/tmp/pymp-gy1ijv6t'
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "/miniconda/lib/python3.7/shutil.py", line 491, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/miniconda/lib/python3.7/shutil.py", line 410, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/miniconda/lib/python3.7/shutil.py", line 406, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/pymp-qe4lk3v5'
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
    finalizer()
OSError: [Errno 24] Too many open files: '/tmp/pymp-erfgy5xu'
  File "/miniconda/lib/python3.7/shutil.py", line 406, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
  File "/miniconda/lib/python3.7/multiprocessing/util.py", line 189, in __call__
    res = self._callback(*self._args, **self._kwargs)
OSError: [Errno 24] Too many open files: '/tmp/pymp-8zwxnw9d'
  File "/miniconda/lib/python3.7/shutil.py", line 491, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/miniconda/lib/python3.7/shutil.py", line 410, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/miniconda/lib/python3.7/shutil.py", line 406, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/pymp-uw_vsn4_'

#0  signal_handler (sig_num=17) at /tmp/build/80754af9/python_1553721932202/work/Modules/signalmodule.c:302
#1  <signal handler called>
#2  0x00007f7caec9c8c2 in do_futex_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00007f7caec9c9d3 in __new_sem_wait_slow () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x000055d691cf5f39 in PyThread_acquire_lock_timed () at /tmp/build/80754af9/python_1553721932202/work/Python/thread_pthread.h:366
#5  0x000055d691d2d84a in acquire_timed (timeout=4999998652, lock=0x7f78d800e4e0) at /tmp/build/80754af9/python_1553721932202/work/Modules/_threadmodule.c:61
#6  lock_PyThread_acquire_lock () at /tmp/build/80754af9/python_1553721932202/work/Modules/_threadmodule.c:144
#7  0x000055d691cfc6e4 in _PyMethodDef_RawFastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:690
#8  0x000055d691cfc86f in _PyMethodDescr_FastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/descrobject.c:288
#9  0x000055d691d5807c in call_function (kwnames=0x0, oparg=3, pp_stack=<synthetic pointer>) at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:4593
#10 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3110
#11 0x000055d691c994f9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3930
#12 0x000055d691cfb9c5 in _PyFunction_FastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:433
#13 0x000055d691d53ad0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:4616
#14 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3110
#15 0x000055d691c994f9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3930
#16 0x000055d691cfba27 in _PyFunction_FastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:433
#17 0x000055d691d548fe in call_function (kwnames=0x7f7c5a49fba8, oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:4616
#18 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3139
#19 0x000055d691c994f9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3930
#20 0x000055d691cfb9c5 in _PyFunction_FastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:433
#21 0x000055d691d53ad0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:4616
#22 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3110
#23 0x000055d691cfb79b in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:283
#24 _PyFunction_FastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:408
#25 0x000055d691d53ad0 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:4616
#26 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3110
#27 0x000055d691c9a50b in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=0x7f7c5a4a6420)
    at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:283
#28 _PyFunction_FastCallDict () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:322
#29 0x000055d691cb1bd0 in _PyObject_FastCall_Prepend () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:866
#30 0x000055d691cf49ba in call_unbound (nargs=0, args=0x0, self=0x7f7c51731160, func=0x7f7c5a438158, unbound=1)
    at /tmp/build/80754af9/python_1553721932202/work/Objects/typeobject.c:1481
#31 call_method (nargs=0, args=0x0, name=0x55d691eb57c0 <PyId___next__.15069>, obj=0x7f7c51731160) at /tmp/build/80754af9/python_1553721932202/work/Objects/typeobject.c:1513
#32 slot_tp_iternext () at /tmp/build/80754af9/python_1553721932202/work/Objects/typeobject.c:6555
#32 slot_tp_iternext () at /tmp/build/80754af9/python_1553721932202/work/Objects/typeobject.c:6555
#33 0x000055d691cc120f in enum_next () at /tmp/build/80754af9/python_1553721932202/work/Objects/enumobject.c:156
#34 0x000055d691d53e76 in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:2809
#35 0x000055d691c99db9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3930
#36 0x000055d691cfba27 in _PyFunction_FastCallKeywords () at /tmp/build/80754af9/python_1553721932202/work/Objects/call.c:433
#37 0x000055d691d53846 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:4616
#38 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3124
#39 0x000055d691c994f9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3930
#40 0x000055d691c9a3c4 in PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:3959
#41 0x000055d691c9a3ec in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /tmp/build/80754af9/python_1553721932202/work/Python/ceval.c:524
#42 0x000055d691db2874 in run_mod () at /tmp/build/80754af9/python_1553721932202/work/Python/pythonrun.c:1035
#43 0x000055d691dbcb81 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1553721932202/work/Python/pythonrun.c:988
#44 0x000055d691dbcd73 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1553721932202/work/Python/pythonrun.c:429
---Type <return> to continue, or q <return> to quit---bt~
#45 0x000055d691dbde5f in pymain_run_file (p_cf=0x7fff7e420f70, filename=0x55d693201fa0 L"train.py", fp=0x55d693248ab0)
    at /tmp/build/80754af9/python_1553721932202/work/Modules/main.c:427
#46 pymain_run_filename (cf=0x7fff7e420f70, pymain=0x7fff7e421080) at /tmp/build/80754af9/python_1553721932202/work/Modules/main.c:1627
#47 pymain_run_python (pymain=0x7fff7e421080) at /tmp/build/80754af9/python_1553721932202/work/Modules/main.c:2877
#48 pymain_main () at /tmp/build/80754af9/python_1553721932202/work/Modules/main.c:3038
#49 0x000055d691dbdf7c in _Py_UnixMain () at /tmp/build/80754af9/python_1553721932202/work/Modules/main.c:3073
#50 0x00007f7cae8bcb97 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#51 0x000055d691d63122 in _start () at ../sysdeps/x86_64/elf/start.S:103
```

My data loader creates a lot of external processes (not a good thing, but I need this for now), but data loader should not hang like that (better to have it crashed).

cc @SsnL