Repository: pytorch
Issue: DistributedDataParallel: `DDP(model)` hangs on non-master node
Labels: oncall: distributed,triaged,module: ddp

## Issue description

I'm using `DistributedDataParallel` to train model in data-parallel distributed mode, here is a simplified, runnable code:

this is my training script:
```python
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import Dataset, IterableDataset, DataLoader
from torch.utils.data.distributed import DistributedSampler
import os
import time


class ToyModel(nn.Module):
    def __init__(self, m):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(m, m, bias=False)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(m, 1, bias=False)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))


def print_dist_info():
    host_name = os.uname()[1]
    pid = os.getpid()
    LOCAL_RANK = os.environ['LOCAL_RANK']
    RANK = os.environ['RANK']
    WORLD_SIZE = os.environ['WORLD_SIZE']
    log_string = f"""
    ===================== {RANK} / {WORLD_SIZE}, {host_name}: {LOCAL_RANK} =====================
    pid = {pid}
    RANK = {RANK}
    WORLD_SIZE = {WORLD_SIZE}
    LOCAL_RANK = {LOCAL_RANK}
    MASTER_ADDR = {os.environ['MASTER_ADDR']}
    MASTER_PORT = {os.environ['MASTER_PORT']}
    """
    print(log_string)


def setup(global_rank, global_world_size, local_rank):
    dist.init_process_group("nccl", rank=global_rank, world_size=global_world_size)
    print(f'=========> dist.init_process_group() done. global_rank={global_rank}, local_rank={local_rank}')


def cleanup(global_rank, global_world_size, local_rank):
    dist.destroy_process_group()
    print(f'=========> dist.destroy_process_group(). global_rank={global_rank}, local_rank={local_rank}')


def main():
    print_dist_info()

    RANK, WORLD_SIZE, LOCAL_RANK = int(os.environ['RANK']), int(os.environ['WORLD_SIZE']), int(os.environ['LOCAL_RANK'])
    setup(RANK, WORLD_SIZE, LOCAL_RANK)
    
    # model
    model = ToyModel(5)

    print('constructing DDP(model) ...')
    gpu_id = LOCAL_RANK
    device = torch.device(gpu_id)
    model.to(device)
    ddp_model = DDP(model, device_ids=[gpu_id])  # <---------- Node1 stick here
    print('constructed DDP(model).')
    
    print('sync...')
    dist.barrier()
    print('sync done.')

    time.sleep(5)
    cleanup(RANK, WORLD_SIZE, LOCAL_RANK)

if __name__ == '__main__':
    main()

```

I use two node, each node 2 process(GPU), and use `torch.distributed.launch` to start processes on each node:

```
# on Node 0: (master)
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="10.1.80.121" --master_port=1240 myscript.py

# on Node 1 (worker)
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="10.1.80.121" --master_port=1240 myscript.py
```

On master node, it works well, console output:

```
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************

    ===================== 1 / 4, emu: 1 =====================
    pid = 14216
    RANK = 1
    WORLD_SIZE = 4
    LOCAL_RANK = 1
    MASTER_ADDR = 10.1.80.121
    MASTER_PORT = 1240


    ===================== 0 / 4, emu: 0 =====================
    pid = 14215
    RANK = 0
    WORLD_SIZE = 4
    LOCAL_RANK = 0
    MASTER_ADDR = 10.1.80.121
    MASTER_PORT = 1240

=========> dist.init_process_group() done. global_rank=1, local_rank=1
=========> dist.init_process_group() done. global_rank=0, local_rank=0
constructing DDP(model) ...
constructing DDP(model) ...
constructed DDP(model).
sync...
constructed DDP(model).
sync...
```

but on Node 1, it hangs on `DDP(model, device_ids=[gpu_id])`

```
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************

    ===================== 3 / 4, ibis: 1 =====================
    pid = 25333
    RANK = 3
    WORLD_SIZE = 4
    LOCAL_RANK = 1
    MASTER_ADDR = 10.1.80.121
    MASTER_PORT = 1240


    ===================== 2 / 4, ibis: 0 =====================
    pid = 25332
    RANK = 2
    WORLD_SIZE = 4
    LOCAL_RANK = 0
    MASTER_ADDR = 10.1.80.121
    MASTER_PORT = 1240

=========> dist.init_process_group() done. global_rank=2, local_rank=0
=========> dist.init_process_group() done. global_rank=3, local_rank=1

constructing DDP(model) ...
constructing DDP(model) ...
```



## System Info

Node 0: (master)

```
Collecting environment information...
PyTorch version: 1.7.1
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.5 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.17.1

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti
GPU 8: GeForce RTX 2080 Ti
GPU 9: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.7.1
[pip3] torchvision==0.8.2
[pip3] torchviz==0.0.1
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] torch                     1.7.1                    pypi_0    pypi
[conda] torchvision               0.8.2                    pypi_0    pypi
[conda] torchviz                  0.0.1                    pypi_0    pypi
```

Node 1:

```
Collecting environment information...
PyTorch version: 1.7.1
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti
GPU 8: GeForce RTX 2080 Ti
GPU 9: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.7.1
[pip3] torchvision==0.8.2
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] torch                     1.7.1                    pypi_0    pypi
[conda] torchvision               0.8.2                    pypi_0    pypi
```

- PyTorch or Caffe2: pytorch
- How you installed PyTorch (conda, pip, source): conda
- Build command you used (if compiling from source):
- OS: ubuntu 18.04
- PyTorch version: 1.7.1
- Python version: 3.8
- CUDA/cuDNN version: 10.2
- GPU models and configuration:
- GCC version (if compiling from source):
- CMake version:
- Versions of any other relevant libraries:


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu