Repository: pytorch
Issue: "distributed" NCCL tests fail when having more than 3 GPUs
Labels: oncall: distributed,module: tests,triaged,module: nccl,module: ddp

## 🐛 Bug

When running the distributed NCCL tests on a system with more than 3 GPUs available the tests fail with a connection refused error.

Tracing that down via ` NCCL_DEBUG=INFO` reveals a warning "bootstrap.cc:190 NCCL WARN Bootstrap Root : mismatch in rank count from procs 6 : 3", which hints that 6 GPUs are available and hence expected but only 3 processes are started which is indeed what happens. See https://github.com/pytorch/pytorch/blob/v1.7.0-rc1/test/run_test.py#L181

As a test I increased the WORLD_SIZE to 6 and the tests succeeded.

## To Reproduce

Steps to reproduce the behavior:

1. Use a system with at least 4 GPUs
1. Run `NCCL_DEBUG="INFO" TEMP_DIR=/tmp/tmp74prol6l BACKEND=nccl INIT_METHOD=env:// WORLD_SIZE=3 TEST_REPORT_SOURCE_OVERRIDE=dist-nccl python distributed/test_distributed_spawn.py --verbose TestDistBackendWithFork.test_DistributedDataParallel` (extracted from `run_test.py`)

Relevant part of the output:

```
taurusml26:67298:67298 [0] NCCL INFO Launch mode Parallel
/tmp/install_pt/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:448: UserWarning: Single-Process Multi-GPU is not the recommended mode for DDP. In this mode, each DDP instance operates on multiple devices and creates multiple module replicas within one process. The overhead of scatter/gather and GIL contention in every forward pass can slow down training. Please consider using one DDP instance per device or per module replica by explicitly setting device_ids or CUDA_VISIBLE_DEVICES. 
  "Single-Process Multi-GPU is not the recommended mode for "
taurusml26:67298:67352 [0] NCCL INFO Channel 00/04 :    0   1
taurusml26:67298:67353 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
taurusml26:67298:67352 [0] NCCL INFO Channel 01/04 :    0   1
taurusml26:67298:67353 [1] NCCL INFO Trees [0] -1/-1/-1->1->0|0->1->-1/-1/-1 [1] 0/-1/-1->1->-1|-1->1->0/-1/-1 [2] -1/-1/-1->1->0|0->1->-1/-1/-1 [3] 0/-1/-1->1->-1|-1->1->0/-1/-1
taurusml26:67298:67352 [0] NCCL INFO Channel 02/04 :    0   1
taurusml26:67298:67353 [1] NCCL INFO Setting affinity for GPU 1 to ffffff,ffffffff,ffffffff
taurusml26:67298:67352 [0] NCCL INFO Channel 03/04 :    0   1
taurusml26:67298:67352 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
taurusml26:67298:67352 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1|-1->0->1/-1/-1 [1] -1/-1/-1->0->1|1->0->-1/-1/-1 [2] 1/-1/-1->0->-1|-1->0->1/-1/-1 [3] -1/-1/-1->0->1|1->0->-1/-1/-1
taurusml26:67298:67352 [0] NCCL INFO Setting affinity for GPU 0 to ffffff,ffffffff,ffffffff
taurusml26:67298:67352 [0] NCCL INFO Channel 00 : 0[404000] -> 1[405000] via P2P/direct pointer
taurusml26:67298:67353 [1] NCCL INFO Channel 00 : 1[405000] -> 0[404000] via P2P/direct pointer
taurusml26:67298:67353 [1] NCCL INFO Channel 01 : 1[405000] -> 0[404000] via P2P/direct pointer
taurusml26:67298:67352 [0] NCCL INFO Channel 01 : 0[404000] -> 1[405000] via P2P/direct pointer
taurusml26:67298:67353 [1] NCCL INFO Channel 02 : 1[405000] -> 0[404000] via P2P/direct pointer
taurusml26:67298:67352 [0] NCCL INFO Channel 02 : 0[404000] -> 1[405000] via P2P/direct pointer
taurusml26:67298:67353 [1] NCCL INFO Channel 03 : 1[405000] -> 0[404000] via P2P/direct pointer
taurusml26:67298:67352 [0] NCCL INFO Channel 03 : 0[404000] -> 1[405000] via P2P/direct pointer
taurusml26:67298:67353 [1] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
taurusml26:67298:67352 [0] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
taurusml26:67298:67353 [1] NCCL INFO comm 0x200400000e00 rank 1 nranks 2 cudaDev 1 busId 405000 - Init COMPLETE
taurusml26:67298:67352 [0] NCCL INFO comm 0x200354000e00 rank 0 nranks 2 cudaDev 0 busId 404000 - Init COMPLETE
taurusml26:67298:67298 [0] NCCL INFO Launch mode Group/CGMD

taurusml26:67298:67367 [0] bootstrap.cc:190 NCCL WARN Bootstrap Root : mismatch in rank count from procs 6 : 3
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying
taurusml26:67300:67371 [4] NCCL INFO Call to connect returned Connection refused, retrying

taurusml26:67300:67371 [4] include/socket.h:403 NCCL WARN Connect to 10.1.148.186<38911> failed : Connection refused
```

## Environment

 - PyTorch Version (e.g., 1.0): 1.7.0-rc1
 - OS (e.g., Linux): Linux

cc @mruberry @VitalyFedyunin @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski