Repository: pytorch
Issue: PyTorch 1.9.1 incorrect result for all-reduce
Labels: oncall: distributed

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

The all-reduce step produces incorrect and inconsistent results across devices for PT 1.9.1 with MPI and NCCL. Whereas PT 1.10.0 directly produces correct and consistent results. 

## To Reproduce

Steps to reproduce the behavior:

1. Pull and run the container from PyTorch dockerhub: __note the container comes with PT 1.10.0, though tagged as PT 1.9.1__
I'm using an instance with 8 GPUs.
```
docker pull pytorch/pytorch:1.9.1-cuda11.1-cudnn8-devel
nvidia-docker run -it pytorch/pytorch:1.9.1-cuda11.1-cudnn8-devel
```
2. install necessary dependencies (for me EFA and MPI):
```
# prepare
cd /
export EFA_VERSION=1.12.1
export EFA_PATH=/opt/amazon/efa
export OMPI_VERSION=4.1.1
export OPEN_MPI_PATH=/opt/amazon/openmpi
apt-get update
apt-get install wget
apt-get install curl
# install EFA
mkdir /tmp/efa \
  && cd /tmp/efa \
  && curl -O https://s3-us-west-2.amazonaws.com/aws-efa-installer/aws-efa-installer-${EFA_VERSION}.tar.gz \
  && tar -xf aws-efa-installer-${EFA_VERSION}.tar.gz \
  && cd aws-efa-installer \
  && ./efa_installer.sh -y --skip-kmod -g \
  && rm -rf $OPEN_MPI_PATH \
  && rm -rf /tmp/efa \
  && rm -rf /tmp/aws-efa-installer-${EFA_VERSION}.tar.gz
# install MPI
mkdir /tmp/openmpi && \
  cd /tmp/openmpi && \
  wget --quiet https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-${OMPI_VERSION}.tar.gz && \
  tar zxf openmpi-${OMPI_VERSION}.tar.gz && \
  cd openmpi-${OMPI_VERSION} && \
  ./configure --enable-orterun-prefix-by-default --prefix=$OPEN_MPI_PATH && \
  make -j $(nproc) all && \
  make install && \
  ldconfig && \
  cd / && \
  rm -rf /tmp/openmpi
# update paths
export PATH="$OPEN_MPI_PATH/bin:$PATH"
export LD_LIBRARY_PATH=$OPEN_MPI_PATH/lib/:$EFA_PATH/lib/:$LD_LIBRARY_PATH
```
3. fetch and run code. The python script can be [found here](https://gist.github.com/ydaiming/29a18796681a1fb510ababffd62c22c3). 
You may want to downgrade PT version first by:
```
pip uninstall torch
pip3 install torch==1.9.1+cu111 -f https://download.pytorch.org/whl/cu111/torch_stable.html
```
Run the script with
```
mpirun --allow-run-as-root --tag-output -np 8  python pt191_bug.py
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

4. The current results from PT 1.9.1 is the following, and the correct result is under section `Expected behavior`.
```
[1,0]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 0.5074, 0.5751, 0.6021], device='cuda:0')
[1,0]<stdout>:Finished nccl
[1,3]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 0.5760, 0.8495, 0.5953], device='cuda:3')
[1,3]<stdout>:Finished nccl
[1,4]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 0.8128, 0.5724, 0.5337], device='cuda:4')
[1,4]<stdout>:Finished nccl
[1,2]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 0.9889, 0.5220, 0.9011], device='cuda:2')
[1,2]<stdout>:Finished nccl
[1,1]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 0.5914, 0.7900, 0.2538], device='cuda:1')
[1,1]<stdout>:Finished nccl
[1,6]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 3.7940, 4.4288, 2.4583], device='cuda:6')
[1,6]<stdout>:Finished nccl
[1,5]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 3.7940, 4.4288, 2.4583], device='cuda:5')
[1,5]<stdout>:Finished nccl
[1,7]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 3.7940, 4.4288, 2.4583], device='cuda:7')
[1,7]<stdout>:Finished nccl
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

Correct and consistent results across devices. The following is by PT 1.10.0.
```
[1,0]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:0')
[1,0]<stdout>:Finished nccl
[1,4]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:4')
[1,4]<stdout>:Finished nccl
[1,3]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:3')
[1,3]<stdout>:Finished nccl
[1,2]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:2')
[1,2]<stdout>:Finished nccl
[1,1]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:1')
[1,1]<stdout>:Finished nccl
[1,7]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:7')
[1,7]<stdout>:Finished nccl
[1,5]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:5')
[1,5]<stdout>:Finished nccl
[1,6]<stdout>:[Result]: tensor([3.6749, 4.9123, 3.6115,  ..., 5.5506, 4.9038, 3.9641], device='cuda:6')
[1,6]<stdout>:Finished nccl
```


## Environment
```
PyTorch version: 1.9.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-1060-aws-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 450.142.00
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.9.1+cu111
[pip3] torchelastic==0.2.0
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py37h7f8727e_0
[conda] mkl_fft                   1.3.1            py37hd3c417c_0
[conda] mkl_random                1.2.2            py37h51133e4_0
[conda] numpy                     1.21.2           py37h20f2e39_0
[conda] numpy-base                1.21.2           py37h79a1101_0
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.9.1+cu111              pypi_0    pypi
[conda] torchelastic              0.2.0                    pypi_0    pypi
[conda] torchtext                 0.11.0                     py37    pytorch
[conda] torchvision               0.11.1               py37_cu111    pytorch
```

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->