Repository: pytorch
Issue: Multi-node training meets unknown error
Labels: oncall: distributed,triaged

### 🐛 Describe the bug

Multi-node training meets unknown error!

The code I use is 

```python
import os
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp

from torch.nn.parallel import DistributedDataParallel as DDP

dist.init_process_group("nccl", rank=int(os.environ["RANK"]), world_size=int(os.environ["WORLD_SIZE"]))
rank = int(os.environ["LOCAL_RANK"])
print(rank)

torch.cuda.set_device(rank)
# create model and move it to GPU with id rank
model = nn.Linear(10, 5).to(rank)
print(model)
ddp_model = DDP(model, device_ids=[rank], output_device=rank)

loss_fn = nn.MSELoss()
optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

for _ in range(10):
    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()
print(ddp_model.module.weight)

dist.destroy_process_group()
```

Then I use the following commands to run:

```
torchrun --nnode 2 --node_rank 0 --nproc_per_node 2 --master_addr 10.1.4.144 --master_port 13579 main.py
torchrun --nnode 2 --node_rank 1 --nproc_per_node 2 --master_addr 10.1.4.144 --master_port 13579 main.py
```

where node 0 ip is 10.1.4.144.

This error occurs on node_rank 0:

```
Traceback (most recent call last):                          
  File "main.py", line 18, in <module>                                                                   
    ddp_model = DDP(model, device_ids=[rank], output_device=rank)
  File "/data/qingsong/anaconda3/envs/th110/lib/python3.8/site-packages/torch/nn/parallel/distributed.py"
, line 578, in __init__
    dist._verify_model_across_ranks(self.process_group, parameters)
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:891, internal error, NCC
L version 21.0.3
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption
Linear(in_features=10, out_features=5, bias=True)
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 20640 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 20639) of bi
nary: /data/qingsong/anaconda3/envs/th110/bin/python
```

While node_rank 1 is hanging.

### Versions

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-163-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 3090
GPU 1: GeForce RTX 3090
GPU 2: GeForce RTX 3090
GPU 3: GeForce RTX 3090
GPU 4: GeForce RTX 3090
GPU 5: GeForce RTX 3090
GPU 6: GeForce RTX 3090
GPU 7: GeForce RTX 3090
GPU 8: GeForce RTX 3090
GPU 9: GeForce RTX 3090

Nvidia driver version: 455.23.04
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.10.1+cu111
[pip3] torchaudio==0.10.1+rocm4.1
[pip3] torchvision==0.11.2+cu111
[conda] numpy                     1.22.4                   pypi_0    pypi
[conda] torch                     1.10.1+cu111             pypi_0    pypi
[conda] torchaudio                0.10.1+rocm4.1           pypi_0    pypi
[conda] torchvision               0.11.2+cu111             pypi_0    pypi

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501