Repository: pytorch
Issue: [torch.distributed.launch|run] Hangs on SIGINT when using a TCPStore backed rdzv_backend 
Labels: oncall: distributed,triaged,module: c10d

## 🐛 Bug

`torch.distributed.launch` and `torch.distributed.run` will not process SIGINT (CTRL+C from CLI) and hang indefinitely without returning control back to the shell when trying to CTRL+C out of the launcher while it is waiting for other nodes to show up (aka during rendezvous)

## To Reproduce

1. (install torch 1.9.0+, issue exists in torch-1.10 nightly as well)
2. Force an incomplete rank with rdzv (`--nnodes 2` but just run the following command once)
    ```shell
     $ LOGLEVEL=INFO python -u -m torch.distributed.run \
                --nproc_per_node 4 \
                --nnodes 2 \
                --master_addr $(hostname) \
                --master_port 6000 \
                ~/test_script.py
    ```
3. (note)  content of `~/test_script.py` does not matter (`torch.distributed.run` gets stuck before the script content is even relevant). But here's what I tested with anyways
    ```py
    import torch.distributed as dist
    from torch.distributed.elastic.multiprocessing.errors import record
    import argparse
    import torch

    @record
    def main():
        import os
        rank = os.environ["RANK"]
        dist.init_process_group("nccl")

    if __name__ == "__main__":
        main()
    ```
4.  #2's command will hang so try CTRL+C'ing it
5. (actual) hangs indefinitely (see screenshot below)

![image](https://user-images.githubusercontent.com/43595115/130516268-fcf6e1b3-fc56-4d7d-8039-5173e2b3988b.png)

6. Sending it SIGINT (`kill -15 $pid`) or SIGKILL (`kill -9 $pid`) from a different terminal works but inconvenient since I lose control over the main terminal

## Expected behavior

CTRL+C should terminate `torch.distributed.run` and return the control back to the main terminal

## Environment

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

PyTorch version: 1.10.0.dev20210823+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.7 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.23

Python version: 3.8.5 (default, Mar 24 2021, 22:32:23)  [GCC 5.4.0 20160609] (64-bit runtime)
Python platform: Linux-4.4.0-1102-aws-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB

Nvidia driver version: 440.33.01
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.2
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] pytorch-lightning==1.3.5
[pip3] torch==1.10.0.dev20210823+cu102
[pip3] torchaudio==0.9.0
[pip3] torchelastic==0.2.3.dev0
[pip3] torchmetrics==0.3.2
[pip3] torchserve==0.4.0
[pip3] torchvision==0.10.0+cu111
[conda] Could not collect

## Additional context

SIGINT seems to be ignored during the creation of TCPStore (which we use for "static" and "c10d" rdzv backends) since the main thread/process is in a tcp accept() call which seems to not be "interruptable". https://stackoverflow.com/questions/44259468/how-to-interrupt-accept-in-a-tcp-ip-server

Using other (etcd) rendezvous backend does not have this issue.



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23