Repository: pytorch
Issue: PyTorch 1.13.1 hangs with `torch.distributed.init_process_group`
Labels: oncall: distributed

### 🐛 Describe the bug

Hi there,
We're upgrading from `PyTorch` `1.12.1` to `1.13.1`, and find that it hangs with `torch.distributed.init_process_group` with `1.13.1` while not `1.12.1`, with the following source code when I turn on `CUDA_LAUNCH_BLOCKING=1`:


```
    def ddp_init(self, core):
        import torch.distributed as dist
        # ...

        init_method = f"tcp://{ip}:{main_port}"
        dist.init_process_group(
            self.cfg.ddp_dist_backend,
            rank=core.rank(),
            world_size=core.size(),
            init_method=init_method,
        )
        dist.barrier()
```

where `self.cfg.ddp_dist_backend = nccl`. 

- We look at https://discuss.pytorch.org/t/torch-distributed-init-process-group-hangs-with-4-gpus-with-backend-nccl-but-not-gloo/149061/1, and `IOMMU` should be `disabled` as:

```
ubuntu@compute-dy-worker-80:~$ sudo lspci -vvv | grep ACSCtl
ubuntu@compute-dy-worker-80:~$ 
```

### Versions

```
root@compute-dy-worker-80:/# python collect_env.py
Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.3
Libc version: glibc-2.31

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-1080-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] sagemaker-pytorch-training==2.7.0
[pip3] torch==1.13.1+cu117
[pip3] torchaudio==0.13.1+cu117
[pip3] torchdata==0.5.1
[pip3] torchnet==0.0.4
[pip3] torchvision==0.14.1+cu117
[conda] Could not collect
root@compute-dy-worker-80:/#
```

```
root@compute-dy-worker-80:/# pip show conda
Name: conda
Version: 22.11.1
Summary: OS-agnostic, system-level binary package manager.
Home-page: https://github.com/conda/conda
Author: Anaconda, Inc.
Author-email: conda@continuum.io
License: BSD-3-Clause
Location: /opt/conda/lib/python3.9/site-packages
Requires: pluggy, pycosat, requests, ruamel.yaml, tqdm
Required-by: mamba
root@compute-dy-worker-80:/#
```

Thank you.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu