Repository: pytorch
Issue: torch.distributed and subprocess do not work together?
Labels: oncall: distributed

## 🐛 Bug

When using `torch.distributed` training, cannot run an additional independent `suprocess` .

I am using `torch.distributed` to run DDP training. Every once in a while I want to start an independent shell command using `subproces.Popen`. This is [not working][1]: the `subprocess.Popen` simply blocks the run without any errors/feedback.

## To Reproduce

This short script reproduces the deadlock:
```
import torch
import os
import subprocess
import torch.distributed as dist
import random


def debug_sp(local_rank, args):
    print(f'running subprocess debug_sp local_rank={local_rank}')

    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = f'{29500 + random.randint(0, 500)}'

    dist.init_process_group(
        backend="nccl",
        init_method='env://',
        world_size=1,
        rank=local_rank,
    )

    cmd = ['python3', '-c', "import time; print('hello world'); time.sleep(5); print('woke up')"]
    print(f'evaluating: {cmd}', flush=True)
    proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    print(f'rank={local_rank}\t{proc}={proc.poll()}')
    try:
        proc_o, proc_e = proc.communicate(timeout=120)
        print(f'rank={local_rank}\tsuccessfully communicated o={proc_o} e={proc_e} poll={proc.poll()}')
    except subprocess.TimeoutExpired:
        proc.kill()
        proc_o, proc_e = proc.communicate()
        print(f'rank={local_rank}\ttime out o={proc_o} e={proc_e} poll={proc.poll()}')


if __name__ == '__main__':
    args = ('dummy', )
    # ready to spawn
    os.environ['WORLD_SIZE'] = '1'
    torch.multiprocessing.spawn(debug_sp, nprocs=2, args=args)
```

Run this script: It should open two subprocess (torch's) and then `subprocess.Popen` from each to run a simple shell script.
However only one of them (on rank 0) is being executed, the second one does not start and blocks the entire execution of all the script:

```
running subprocess debug_sp local_rank=0
running subprocess debug_sp local_rank=1
evaluating: ['python3', '-c', "import time; print('hello world'); time.sleep(5); print('woke up')"]
rank=0  <subprocess.Popen object at 0x2b95242235f8>=None
rank=0  successfully communicated o=b'hello world\nwoke up\n' e=b'' poll=0
```
## Expected behavior
I expect `subprocess.Popen` to work both on rank=0 and rank=1 subprocesses and I do not expect to see any deadlock in this case.


## Environment

```
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.9

Python version: 3.6.13 |Anaconda, Inc.| (default, Feb 23 2021, 21:15:04)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-centos-7.8.2003-Core
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 455.45.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.8.0
[pip3] torch-scatter==2.0.7
[pip3] torchaudio==0.8.0a0+a751e1d
[pip3] torchtext==0.9.0
[pip3] torchvision==0.9.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] faiss-cpu                 1.6.0            py36h6bb024c_0    pytorch
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py36he8ac12f_0
[conda] mkl_fft                   1.3.0            py36h54f3939_0
[conda] mkl_random                1.1.1            py36h0573a6f_0
[conda] numpy                     1.19.2           py36h54aff64_0
[conda] numpy-base                1.19.2           py36hfa32c7d_0
[conda] pytorch                   1.8.0           py3.6_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-scatter           2.0.7           py36_torch_1.8.0_cu102    rusty1s
[conda] torchaudio                0.8.0                      py36    pytorch
[conda] torchtext                 0.9.0                      py36    pytorch
[conda] torchvision               0.9.0                py36_cu102    pytorch
```

 [1]: https://stackoverflow.com/q/68516184/1714410

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23