Repository: pytorch
Issue: Inconsistent multi-node latency with NCCL and OpenMPI
Labels: oncall: distributed,module: nccl,module: openmp

### 🐛 Describe the bug

I deployed PyTorch on 2 servers (with 1 GPU each), and I am trying to measure the communication latency using the following codes, which simply execute AllReduce operation for multiple times and calculate the average time spent. (The process group is initialized with OpenMPI.)
```python
import argparse
import time
import numpy as np
import torch
import torch.distributed as dist
from mpi4py import MPI

def run(vector_size, rank, steps):
    elapsedTime = 0
    for step in range(1, steps + 1):

        # compute reduced sum
        tensor = torch.randint(10,(vector_size,)).cuda()

        start = time.monotonic_ns()
        dist.all_reduce(tensor, op=dist.ReduceOp.SUM, async_op=False)
        latency = (time.monotonic_ns() - start) / 1e3

        elapsedTime += latency

        #time.sleep(0.1)

    elapsedTime/=steps
    if rank==0:
        print(vector_size*4, elapsedTime)
    


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--backend', type=str, default='nccl', help='Name of the backend to use.')
    parser.add_argument(
        '-i',
        '--init-method',
        type=str,
        default='tcp://127.0.0.1:23456',
        help='URL specifying how to initialize the package.')
    parser.add_argument('-s', '--world-size', type=int, help='Number of processes participating in the job.')
    parser.add_argument('--steps', type=int, default=20)
    args = parser.parse_args()
    print(args)

    comm = MPI.COMM_WORLD
    dist.init_process_group(
        backend=args.backend,
        init_method=args.init_method,
        world_size=args.world_size,
        rank=comm.Get_rank(),
    )

    print("size(B)","latency(us)")
    vector_size=2
    rank=comm.Get_rank()
    for _ in range(20):
        run(vector_size, rank, args.steps)
        vector_size*=2


if __name__ == '__main__':
    main()
```
I ran the code with commands like this:
```
~/conda_env/bin/mpirun np 2 -N 1 --host 11.0.0.1,11.0.0.2 ~/conda_env/bin/python script.py -s 2 -i tcp://11.0.0.1:23456
```
I found the measured latency abnormally high with PyTorch 1.10 + NCCL 2.10:
```
size(B)      latency(us)
8            826.9433000000001
16           908.8419
32           1479.80385
64           2279.2819499999996
128          504.1064
256          1348.6622499999999
512          1123.6129000000003
1024         2590.2159
2048         1715.0593000000001
4096         5227.415999999999
8192         3131.40595
16384        3009.81275
32768        1614.2130499999998
65536        6010.794950000001
131072       6169.70775
262144       6595.7269
524288       4651.931450000001
1048576      5800.938
2097152      7393.041899999999
```

However, if I add ```time.sleep(0.1)``` at the end of each iteration of the function `run`, the latency becomes much smaller:
```
size(B)      latency(us)
8            153.83099999999996
16           157.3773
32           157.008
64           157.7295
128          140.99030000000002
256          130.14204999999998
512          107.28104999999998
1024         117.73960000000002
2048         87.42374999999997
4096         86.94415000000002
8192         110.07860000000001
16384        116.90845000000002
32768        224.9045
65536        113.87135
131072       409.87255000000016
262144       370.2254
524288       837.1048000000001
1048576      1105.72925
2097152      3323.8366499999997
```
The inconsistency also happens between different versions of NCCL. I have recompiled PyTorch with the official build of NCCL 2.11, which has similar results. However, with NCCL 2.7 instead, the latency is always small regardless the interval. The interval does not impact the latency of GLOO, either.

The inconsistency occurs only when we use `mpirun`. If we start the script on each host manually, the latency is always the smaller value. 

What may be the reason of these different latency values? And what is the correct way to measure the performance of AllReduce operations in PyTorch?


### Versions

#### Output of `collect_env.py`
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-6ubuntu2) 7.5.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:23:11)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-74-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA Tesla V100-PCIE-16GB
Nvidia driver version: 465.19.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.4
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0
[pip3] torchvision==0.11.1
[conda] Could not collect

#### Other information
GPUDirect not enabled
Network Interface: Mellanox mlx5 (We use RoCEv2 for NCCL)
MPI version: OpenMPI 4.1.2

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang