Repository: pytorch
Issue: How do I debug "RuntimeError: trying to initialize the default process group twice!"
Labels: oncall: distributed,triaged

## 🐛 Bug

When triggering distributed training in pytorch, the error `RuntimeError: trying to initialize the default process group twice!` occurs. How would one debug it?

## To Reproduce

Steps to reproduce the behavior:

1. on master node ip 10.163.60.19, run `local_rank=0; master_port=1303; python -m torch.distributed.launch --node_rank=$local_rank --master_addr="10.163.60.19" --master_port=$master_port regression_train.py --config_file weights/cnn3_pad_32_pad_ratio_5.ini --distributed 1 --local_rank $local_rank --master_addr 10.163.60.19 --master_port $master_port --world_size 2`
2. on slave node ip 10.163.60.18, run `local_rank=1; master_port=1303; python -m torch.distributed.launch --node_rank=$local_rank --master_addr="10.163.60.19" --master_port=$master_port regression_train.py --config_file weights/cnn3_pad_32_pad_ratio_5.ini --distributed 1 --local_rank $local_rank --master_addr 10.163.60.19 --master_port $master_port --world_size 2`

```
Traceback (most recent call last):
  File "regression_train.py", line 180, in <module>
    torch.distributed.init_process_group(backend=args.distributed_backend, world_size=args.world_size)
  File "/home/kdang/anaconda3/envs/edge-detection/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 370, in init_process_group
    raise RuntimeError("trying to initialize the default process group "
RuntimeError: trying to initialize the default process group twice!
Traceback (most recent call last):
  File "/home/kdang/anaconda3/envs/edge-detection/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kdang/anaconda3/envs/edge-detection/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kdang/anaconda3/envs/edge-detection/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/kdang/anaconda3/envs/edge-detection/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/kdang/anaconda3/envs/edge-detection/bin/python', '-u', 'regression_train.py', '--local_rank=0', '--config_file', 'weights/cnn3_pad_32_pad_ratio_5.ini', '--distributed', '1', '--local_rank', '0', '--master_addr', '10.163.60.19', '--master_port', '1303', '--world_size', '2']' returned non-zero exit status 1.
```

## Expected behavior

Distributed training should start

## Environment

```
(edge-detection) ➜  EDGE-DETECTION-TRAINER git:(master) python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080
Nvidia driver version: 418.152.00
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.4

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.4.0
[pip] torchvision==0.5.0
[conda] mkl                       2020.0                      166    conda-forge
[conda] pytorch                   1.4.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.5.0                py37_cu101    pytorch
(edge-detection) ➜  EDGE-DETECTION-TRAINER git:(master)
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski