Repository: pytorch
Issue: Pytorch with CUDA aware OpenMPI for Infiniband not working with HCOLL and MXM
Labels: oncall: distributed,triaged,module: c10d,distributed-backlog

## Issue description
Dear all,

I try to build PyTorch with CUDA aware OpenMPI working with Infiniband. I'm using a Mellanox Infiniband card. When running this test script
```
$ cat scatter-min
#!/usr/bin/env python
import numpy as np
import torch as to
from torch import distributed as dist

# Initialization
dist.init_process_group('mpi')
rank = dist.get_rank()
world_size = dist.get_world_size()

gpu = 1
D = 5
N = 50
N_reshape = int(50 / 5)
N_chunk = int(N_reshape / world_size)

no_GPUs = to.cuda.device_count()
if gpu:
    dtype_f = to.cuda.FloatTensor 
    to.cuda.set_device(0)
else:
    dtype_f = to.FloatTensor    

dist.barrier()

# Create some array to be split up and scattered
if rank == 0:
    A = to.tensor(np.arange(N).reshape(N_reshape, D)).type(dtype_f)
    A_chunk = to.chunk(A, world_size, dim=0)
    print("Array to be scattered from rank 0 is \n%s" % (A))
else:
    A_chunk = to.tensor([])
    
dist.barrier()

my_A = to.zeros(N_chunk, D).type(dtype_f)
for r in range(world_size):    
    if r == rank:
        print("Before Scatter: Rank %i has \n%s" % (rank, my_A))
    dist.barrier()

# Scatter data into my_A arrays
dist.scatter(my_A, src=0, scatter_list=list(A_chunk))

for r in range(world_size):    
    if r == rank:
        print("After Scatter: Rank %i has \n%s" % (rank, my_A))
    dist.barrier()
```
I run into a fatal error with the output (mind the execution has to be with -n 2)
```
$ mpirun -n 2 ./scatter-min 
Array to be scattered from rank 0 is 
tensor([[ 0.,  1.,  2.,  3.,  4.],
        [ 5.,  6.,  7.,  8.,  9.],
        [10., 11., 12., 13., 14.],
        [15., 16., 17., 18., 19.],
        [20., 21., 22., 23., 24.],
        [25., 26., 27., 28., 29.],
        [30., 31., 32., 33., 34.],
        [35., 36., 37., 38., 39.],
        [40., 41., 42., 43., 44.],
        [45., 46., 47., 48., 49.]], device='cuda:0')
Before Scatter: Rank 0 has 
tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]], device='cuda:0')
Before Scatter: Rank 1 has 
tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]], device='cuda:0')
[gold04:14866:0] Caught signal 11 (Segmentation fault)
==== backtrace ====
 2 0x000000000006858c mxm_handle_error()  /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel7-u4-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.1.0-gcc-MLNX_OFED_LINUX-4.3-1.0.1.0-redhat7.4-x86_64/mxm-hpcx-v2.1/src/mxm/util/debug/debug.c:641
 3 0x0000000000068adc mxm_error_signal_handler()  /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel7-u4-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.1.0-gcc-MLNX_OFED_LINUX-4.3-1.0.1.0-redhat7.4-x86_64/mxm-hpcx-v2.1/src/mxm/util/debug/debug.c:616
 4 0x0000000000036280 killpg()  ??:0
 5 0x000000000015651a __memcpy_ssse3_back()  :0
 6 0x000000000005ac4e mxm_proto_set_data_buf_inline()  /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel7-u4-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.1.0-gcc-MLNX_OFED_LINUX-4.3-1.0.1.0-redhat7.4-x86_64/mxm-hpcx-v2.1/src/mxm/proto/proto_ops.c:271
 7 0x0000000000070085 mxm_shm_channel_send()  /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel7-u4-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.1.0-gcc-MLNX_OFED_LINUX-4.3-1.0.1.0-redhat7.4-x86_64/mxm-hpcx-v2.1/src/mxm/tl/shm/shm_channel.c:321
 8 0x0000000000062030 mxm_proto_conn_send_op()  /scrap/jenkins/workspace/hpc-power-pack/label/r-vmb-rhel7-u4-x86-64-MOFED-CHECKER/hpcx_root/src/hpcx-v2.1.0-gcc-MLNX_OFED_LINUX-4.3-1.0.1.0-redhat7.4-x86_64/mxm-hpcx-v2.1/src/mxm/proto/proto_send.c:47
 9 0x000000000017e2ed mca_pml_yalla_send()  /usr/local/src/openmpi/ompi/mca/pml/yalla/pml_yalla.c:514
10 0x00000000000abfbb ompi_coll_base_scatter_intra_basic_linear()  /usr/local/src/openmpi/ompi/mca/coll/base/coll_base_scatter.c:243
11 0x000000000009339b PMPI_Scatter()  //usr/local/src/openmpi/ompi/mpi/c/profile/pscatter.c:164
12 0x000000000063a670 _ZNSt17_Function_handlerIFvRSt10unique_ptrIN4c10d9WorkEntryESt14default_deleteIS2_EEEZNS1_15ProcessGroupMPI7scatterERSt6vectorIN2at6TensorESaISB_EERS9_ISD_SaISD_EERKNS1_14ScatterOptionsEEUlS6_E_E9_M_invokeERKSt9_Any_dataS6_()  ProcessGroupMPI.cpp:0
13 0x0000000000637655 _ZN4c10d15ProcessGroupMPI7runLoopEv()  ??:0
14 0x00000000000b8678 execute_native_thread_routine_compat()  /opt/conda/conda-bld/compilers_linux-64_1534514838838/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/src/c++11/thread.cc:94
15 0x0000000000007dd5 start_thread()  pthread_create.c:0
16 0x00000000000feb3d __clone()  ??:0
===================
```
When gpu = 0 in line 11 of the test script scatter-min, it runs as expected
```
$ mpirun -n 2 ./scatter-min 
Array to be scattered from rank 0 is 
tensor([[ 0.,  1.,  2.,  3.,  4.],
        [ 5.,  6.,  7.,  8.,  9.],
        [10., 11., 12., 13., 14.],
        [15., 16., 17., 18., 19.],
        [20., 21., 22., 23., 24.],
        [25., 26., 27., 28., 29.],
        [30., 31., 32., 33., 34.],
        [35., 36., 37., 38., 39.],
        [40., 41., 42., 43., 44.],
        [45., 46., 47., 48., 49.]])
Before Scatter: Rank 0 has 
tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])
Before Scatter: Rank 1 has 
tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])
After Scatter: Rank 0 has 
tensor([[ 0.,  1.,  2.,  3.,  4.],
        [ 5.,  6.,  7.,  8.,  9.],
        [10., 11., 12., 13., 14.],
        [15., 16., 17., 18., 19.],
        [20., 21., 22., 23., 24.]])
After Scatter: Rank 1 has 
tensor([[25., 26., 27., 28., 29.],
        [30., 31., 32., 33., 34.],
        [35., 36., 37., 38., 39.],
        [40., 41., 42., 43., 44.],
        [45., 46., 47., 48., 49.]])
```
This is only happening, when OpenMPI is compiled with MXM and HCOLL for CUDA. UCX and KNEM only seems to be sufficient already. OpenMPI was build the following way
```
# Adapt contrib/platform/mellanox/optimized, otherwise libopen-pla.la is has no make rule
$ sed -i '/with_devel_headers/c with_devel_headers=no' /cluster/src/programs/openmpi/3.0.0/contrib/platform/mellanox/optimized

# using environment modules pointing to my locally installed and compiled software
$ module add mellanox_hpc-x/4.3-1.0.1.0/knem mellanox_hpc-x/4.3-1.0.1.0/ucx-cuda mellanox_hpc-x/4.3-1.0.1.0/hcoll-cuda mellanox_hpc-x/4.3-1.0.1.0/mxm gcc/4.8.5 cuda/9.1.85 nccl/2.1.15-cuda-9.1 libfabric/1.6 valgrind/3.13.0 zlib/1.2.8 libibmad/1.3.9
$ mellanox_autodetect=yes mellanox_threads=yes ./configure --prefix=/usr/local/openmpi --localstatedir=/var --with-platform=contrib/platform/mellanox/optimized --enable-static --with-zlib=/usr/local/zlib --with-cuda=/usr/local/cuda-9.1 --with-jdk-dir=/usr/local/java --with-slurm=/usr/local/slurm --with-pmi=/usr/local/slurm --with-valgrind=/usr/local/valgrind --with-libfabric
[...]
Open MPI configuration:
-----------------------
Version: 3.0.0
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi
MPI Build Java bindings (experimental): no
Build Open SHMEM support: yes
Debug build: no
Platform file: contrib/platform/mellanox/optimized

Miscellaneous
-----------------------
CUDA support: yes
 
Transports
-----------------------
Cray uGNI (Gemini/Aries): no
Intel Omnipath (PSM2): no
Intel SCIF: no
Intel TrueScale (PSM): no
Mellanox MXM: yes
Open UCX: yes
OpenFabrics Libfabric: yes
OpenFabrics Verbs: yes
Portals4: no
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: yes
Shared memory/XPMEM: no
TCP: yes
 
Resource Managers
-----------------------
Cray Alps: no
Grid Engine: no
LSF: no
Moab: no
Slurm: yes
ssh/rsh: yes
Torque: no
[...]
$ make -j all
$ make check
# No errors
$ make install
```
Compiling mpi4py and installing it following the pytorch build command results in
```
$ module add openmpi/3.0.0
$ TORCH_CUDA_ARCH_LIST="6.1;6.0;5.2;3.5" USE_CUDNN=1 USE_DISTRIBUTED=1 USE_FFMPEG=1 NO_MKLDNN=1 CUDA_HOME=$CUDA_ROOT python setup.py install
[...]
-- ******** Summary ********
-- General:
--   CMake version         : 3.12.2
--   CMake command         : /usr/local/python/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/local/gcc/bin/c++
--   C++ compiler version  : 4.8.5
--   BLAS                  : MKL
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_NAMESPACE=onnx_torch;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
--   CMAKE_PREFIX_PATH     : /usr/local/python/lib/python2.7/site-packages
--   CMAKE_INSTALL_PREFIX  : /usr/local/src/pytorchtorch/lib/tmp_install
-- 
--   TORCH_VERSION         : 1.0.0
--   CAFFE2_VERSION        : 1.0.0
--   BUILD_ATEN_MOBILE     : OFF
--   BUILD_ATEN_ONLY       : OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.15
--     Python executable   : /usr/local/python/bin/python
--     Pythonlibs version  : 2.7.15
--     Python library      : /usr/local/python/lib/libpython2.7.so.1.0
--     Python includes     : /usr/local/python/include/python2.7
--     Python site-packages: lib/python2.7/site-packages
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : ON
--   USE_ASAN              : OFF
--   USE_CUDA              : 1
--     CUDA static link    : 0
--     USE_CUDNN           : ON
--     CUDA version        : 9.1
--     cuDNN version       : 7.4.1
--     CUDA root directory : /usr/local/cuda-9.1
--     CUDA library        : /usr/lib64/libcuda.so
--     cudart library      : /usr/local/cuda-9.1/lib64/libcudart_static.a;-pthread;dl;/usr/lib64/librt.so
--     cublas library      : /usr/local/cuda-9.1/lib64/libcublas.so;/usr/local/cuda-9.1/lib64/libcublas_device.a
--     cufft library       : /usr/local/cuda-9.1/lib64/libcufft.so
--     curand library      : /usr/local/cuda-9.1/lib64/libcurand.so
--     cuDNN library       : /usr/local/cudnn7.4-cuda9.1lib64/libcudnn.so.7
--     nvrtc               : /usr/local/cuda-9.1/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda-9.1/include
--     NVCC executable     : /usr/local/cuda-9.1/bin/nvcc
--     CUDA host compiler  : /usr/local/gcc/bin/cc
--     USE_TENSORRT        : OFF
--   USE_ROCM              : 0
--   USE_EIGEN_FOR_BLAS    : 
--   USE_FBGEMM            : 0
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : OFF
--   USE_MOBILE_OPENGL     : OFF
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : OFF
--   USE_NNPACK            : 1
--   USE_NUMPY             : OFF
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : 1
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI             : ON
--     USE_GLOO            : ON
--     USE_GLOO_IBVERBS    : 1
--   Public Dependencies  : Threads::Threads;caffe2::mkl
--   Private Dependencies : qnnpack;nnpack;cpuinfo;fp16;/usr/local/openmpi/lib/libmpi.so;gloo;aten_op_header_gen;onnxifi_loader;rt;gcc_s;gcc;dl
-- Configuring done
[...]
```
with no error. As I said, when leaving out hcoll/cuda and mxm, scatter-min runs with gpu = 1. Is MXM and HCOLL/CUDA not supported by pytorch? Is this not necessary for a pytorch installation with CUDA aware OpenMPI for Infiniband? Hope something can answer this question, I know it's kind of specific.

Looking forward to your answers and best wishes,
fwillo

## System Info
```
$ python collect_env.py 
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server 7.4 (Maipo)
GCC version: (GCC) 4.8.5
CMake version: version 3.12.2

Python version: 2.7
Is CUDA available: N/A
CUDA runtime version: 9.1.85
GPU models and configuration: 
GPU 0: Tesla P100-PCIE-16GB
GPU 1: Tesla P100-PCIE-16GB

Nvidia driver version: 390.12
cuDNN version: Could not collect (Should be v7.4.1.5 for cuda-9.1.85)

Versions of relevant libraries:
[pip] Could not collect
[conda] blas                      1.0                         mkl  
[conda] magma-cuda91              2.3.0                         1    pytorch
[conda] mkl                       2019.1                      144  
[conda] mkl-include               2019.1                      144  
[conda] mkl-rt                    11.1                         p0  
[conda] mkl-service               1.1.0                   py27_p0  
[conda] mkldnn                    0.16.1                        0    mingfeima
[conda] numpy                     1.9.3                   py27_p0  [mkl]
[conda] scipy                     0.16.1              np19py27_p0  [mkl]
[conda] torch                     1.0.0a0+db5d313           <pip>
```


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski