Repository: pytorch
Issue: RuntimeError: broken pipe from NCCL
Labels: oncall: distributed,triaged,module: nccl

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

1. Run this script:
```
import os
import torch

world_size = 2

def _do_useless_allreduce(group):
    # doing this seems to avoid "RuntimeError: Broken pipe" when using a group with a set 
    # of ranks that does not include rank 0
    x = torch.ones((1,), dtype=torch.float32, device="cuda")
    torch.distributed.all_reduce(x, group=group)


def run(gpu):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    torch.cuda.set_device(gpu)
    print(f"rank = {gpu}")

    torch.distributed.init_process_group(
        backend='nccl',
        init_method='env://',
        world_size=world_size,
        rank=gpu)

    # works
    # group = torch.distributed.new_group(ranks=(0,))
    # print(f"BARRIER UP -> GPU:{gpu}")
    # torch.distributed.barrier(group=group)
    # print(f"BARRIER DOWN -> GPU:{gpu}")

    # fails with RuntimeError: broken pipe
    group = torch.distributed.new_group(ranks=(1,))
    print(f"BARRIER UP -> GPU:{gpu}")
    torch.distributed.barrier(group=group)
    print(f"BARRIER DOWN -> GPU:{gpu}")

    # works
    # _do_useless_allreduce(torch.distributed.group.WORLD)
    # group = torch.distributed.new_group(ranks=(1,))
    # _do_useless_allreduce(group)
    # print(f"BARRIER UP -> GPU:{gpu}")
    # torch.distributed.barrier(group=group)
    # print(f"BARRIER DOWN -> GPU:{gpu}")

def main():
    torch.multiprocessing.spawn(run, nprocs=world_size, join=True)


if __name__ == '__main__':
    main()
```

This crashes:

```
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/root/test.py", line 34, in run
    torch.distributed.barrier(group=group)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1487, in barrier
    work = group.barrier()
RuntimeError: Broken pipe
```

## Expected behavior

No error.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

PyTorch version: 1.5.1+cu101
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.11.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
GPU 4: Tesla V100-SXM2-32GB
GPU 5: Tesla V100-SXM2-32GB
GPU 6: Tesla V100-SXM2-32GB
GPU 7: Tesla V100-SXM2-32GB

Nvidia driver version: 418.87.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip] numpy==1.18.5
[pip] torch==1.5.1+cu101
[pip] torchvision==0.6.1+cu101
[conda] numpy                     1.16.3                   pypi_0    pypi
[conda] torch                     1.5.1+cu101              pypi_0    pypi
[conda] torchvision               0.6.1+cu101              pypi_0    pypi

## Additional context

This looks to be the same issue but was closed: https://github.com/pytorch/pytorch/issues/39722


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse