Repository: pytorch
Issue: Torch.multiprocessing.spawn can deadlock
Labels: module: multiprocessing,triaged

## 🐛 Bug

Currently, when the error occurs on one of the child processes, torch.mp.spawn tries to terminate all alive processes and then waits for them to finish: https://github.com/pytorch/pytorch/blob/master/torch/multiprocessing/spawn.py#L99

The code can be stuck on this line, even when the user passes the timeout=0 to the spawn method. This can happen when child processes have registered termination handlers that are stuck forever.


## To Reproduce

The code to reproduce the issue:

```
import atexit
import os
import signal
import time

from torch.multiprocessing.spawn import spawn


def exit_gracefully(p1=None, p2=None):
    print(f"start exiting gracefully pid: {os.getpid()}")
    while True:
        pass
    print(f"finish exiting gracefully pid: {os.getpid()}")


def child_method(idx=0):
    print(f"starting child process: {idx} pid: {os.getpid()}")
    if idx != 1 and idx != 2:
        signal.signal(signal.SIGTERM, exit_gracefully)
        atexit.register(exit_gracefully)
    else:
        raise Exception(f"Error from child process: {idx} pid: {os.getpid()}")


def main_process():
    ctx = spawn(child_method, args=(), nprocs=5, daemon=False, join=False)
    while not ctx.join(-1):
        print("waiting")
        time.sleep(1)
        pass


if __name__ == "__main__":
    main_process()
```


## Expected behavior

The expected behavior should be torch.multiprocessing.spawn follows the timeout argument and does not deadlock.

## Environment

Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Mac OSX 10.15.2
GCC version: Could not collect
CMake version: version 3.17.2

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.18.1
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      233
[conda] mkl-service               2.3.0            py37hfbe908c_0
[conda] mkl_fft                   1.0.14           py37h5e564d8_0
[conda] mkl_random                1.1.0            py37ha771720_0
[conda] numpy                     1.17.2           py37h99e6662_0
[conda] numpy-base                1.17.2           py37h6575580_0
[conda] numpydoc                  0.9.1                      py_0
[conda] pytorch-sphinx-theme      0.0.24                    dev_0    <develop>
[conda] torch                     1.5.0                    pypi_0    pypi
[conda] torchelastic              0.2.0                    pypi_0    pypi
[conda] torchvision               0.6.0                    pypi_0    pypi
## Additional context

What if we add the timeout to the `process.join()` method call and if the process is still alive after that, we send the SIGKILL?
E.g.:

```
for process in self.processes:
    if process.is_alive():
        process.terminate()
    process.join(timeout)
    if process.is_alive():
        process.kill()
```

Please let me know if the deadlock is not expected behavior so I can submit a pull request with the proposed fix.