Repository: pytorch
Issue: Undefined reference to ncclCommSplit
Labels: oncall: distributed,module: nccl

### 🐛 Describe the bug

Building Pytorch from source (main branch) with MPI is giving undefined reference to ncclCommSplit since 1 week. NCCL version is 2.18.3.

Complete error:
```
[6498/6931] Linking CXX shared library lib/libtorch_cuda.so
Warning: Unused direct dependencies:
	/opt/hpcx/ompi/lib/libmpi.so.40
[6499/6931] Building CXX object test_edge_op_registration/CMakeFiles/unbox_lib.dir/custom_ops.cpp.o
[6500/6931] Building CXX object test_edge_op_registration/CMakeFiles/unbox_lib.dir/__/out/RegisterCodegenUnboxedKernelsEverything.cpp.o
[6501/6931] Linking CXX shared library lib/libtorch.so
Warning: Unused direct dependencies:
	/pytorch/build/lib/libtorch_cuda.so
[6502/6931] Building CXX object caffe2/CMakeFiles/backend_fallback_test.dir/__/aten/src/ATen/core/dispatch/backend_fallback_test.cpp.o
[6503/6931] Linking CXX executable bin/CppSignature_test
FAILED: bin/CppSignature_test 
: && /opt/rh/devtoolset-9/root/usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -Wl,--no-as-needed -rdynamic caffe2/CMakeFiles/CppSignature_test.dir/__/aten/src/ATen/core/dispatch/CppSignature_test.cpp.o -o bin/CppSignature_test -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/pytorch/build/lib:/usr/local/cuda-11.8/lib64  lib/libgtest_main.a  -Wl,--no-as-needed,"/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/intel/lib/libmkl_intel_lp64.a  /opt/intel/lib/libmkl_gnu_thread.a  /opt/intel/lib/libmkl_core.a  -fopenmp  -lpthread  /lib64/libm.so  /lib64/libdl.so  -Wl,--no-as-needed,"/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda-11.8/lib64/libcudart_static.a  -ldl  /lib64/librt.so  /usr/local/cuda-11.8/lib64/libnvToolsExt.so  lib/libgtest.a  -pthread && /usr/local/lib64/python3.6/site-packages/cmake/data/bin/cmake -E __run_co_compile --lwyu=bin/CppSignature_test && :
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /pytorch/build/lib/libtorch_cuda.so: undefined reference to `ncclCommSplit'
collect2: error: ld returned 1 exit status
[6504/6931] Building CXX object caffe2/CMakeFiles/kernel_function_legacy_test.dir/__/aten/src/ATen/core/boxing/impl/kernel_function_legacy_test.cpp.o
[6505/6931] Building CXX object caffe2/CMakeFiles/make_boxed_from_unboxed_functor_test.dir/__/aten/src/ATen/core/boxing/impl/make_boxed_from_unboxed_functor_test.cpp.o
[6506/6931] Linking CXX executable bin/parallel_benchmark
FAILED: bin/parallel_benchmark 
: && /opt/rh/devtoolset-9/root/usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -Wl,--no-as-needed -rdynamic test_api/CMakeFiles/parallel_benchmark.dir/parallel_benchmark.cpp.o -o bin/parallel_benchmark -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/pytorch/build/lib:/usr/local/cuda-11.8/lib64  lib/libtorch.so  -Wl,--no-as-needed,"/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/intel/lib/libmkl_intel_lp64.a  /opt/intel/lib/libmkl_gnu_thread.a  /opt/intel/lib/libmkl_core.a  -fopenmp  -lpthread  /lib64/libm.so  /lib64/libdl.so  -Wl,--no-as-needed,"/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda-11.8/lib64/libcudart_static.a  -pthread  -ldl  /lib64/librt.so  /usr/local/cuda-11.8/lib64/libnvToolsExt.so && /usr/local/lib64/python3.6/site-packages/cmake/data/bin/cmake -E __run_co_compile --lwyu=bin/parallel_benchmark && :
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /pytorch/build/lib/libtorch_cuda.so: undefined reference to `ncclCommSplit'
collect2: error: ld returned 1 exit status
[6507/6931] Building CXX object caffe2/CMakeFiles/cpu_rng_test.dir/__/aten/src/ATen/test/cpu_rng_test.cpp.o
[6508/6931] Building CXX object test_edge_op_registration/CMakeFiles/unbox_lib.dir/operator_registry.cpp.o
[6509/6931] Linking CXX shared library lib/libtorch_cuda_linalg.so
Warning: Unused direct dependencies:
	/usr/local/cuda-11.8/lib64/libnvToolsExt.so.1
[6510/6931] Building CXX object test_jit/CMakeFiles/jitbackend_test.dir/test_backend_lib.cpp.o
[6511/6931] Building CUDA object caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/cuda_optional_test.cu.o
[6512/6931] Building CXX object test_jit/CMakeFiles/torchbind_test.dir/test_custom_class_registrations.cpp.o
[6513/6931] Building CUDA object caffe2/CMakeFiles/cuda_vectorized_test.dir/__/aten/src/ATen/test/cuda_vectorized_test.cu.o
[6514/6931] Building CUDA object caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/cuda_half_test.cu.o
../aten/src/ATen/test/cuda_half_test.cu(36): warning #177-D: variable "threshold" was declared but never referenced
../aten/src/ATen/test/cuda_distributions_test.cu(122): warning #177-D: variable "dim_block" was declared but never referenced

ninja: build stopped: subcommand failed.
Building wheel torch-2.2.0.dev20231130+cu118
-- Building version 2.2.0.dev20231130+cu118
cmake -GNinja -DBUILD_LIBTORCH_CPU_WITH_DEBUG=0 -DBUILD_PYTHON=True -DBUILD_SPLIT_CUDA=ON -DBUILD_TEST=True -DCAFFE2_STATIC_LINK_CUDA=1 -DCMAKE_ARGS= -DCMAKE_BUILD_TYPE=Release -DCMAKE_INCLUDE_PATH=/opt/openssl/include:/opt/intel/include: -DCMAKE_INSTALL_PREFIX=/pytorch/torch -DCMAKE_LIBRARY_PATH=/opt/intel/lib:/lib: -DCMAKE_PREFIX_PATH=/opt/python/cp38-cp38/lib/python3.8/site-packages -DGLIBCXX_USE_CXX11_ABI=0 -DINSTALL_TEST=0 -DNUMPY_INCLUDE_DIR=/opt/python/cp38-cp38/lib/python3.8/site-packages/numpy/core/include -DOPENSSL_ROOT_DIR=/opt/openssl -DPYTHON_EXECUTABLE=/opt/python/cp38-cp38/bin/python -DPYTHON_INCLUDE_DIR=/opt/python/cp38-cp38/include/python3.8 -DPYTHON_LIBRARY=/opt/_internal/cpython-3.8.1/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=2.2.0.dev20231130+cu118 -DTORCH_CUDA_ARCH_LIST=5.0;6.0;7.0;7.5;8.0;8.6;3.7;9.0 -DUSE_CUPTI_SO=0 -DUSE_CUSPARSELT=1 -DUSE_DISTRIBUTED=1 -DUSE_KINETO=1 -DUSE_MPI=1 -DUSE_NCCL=1 -DUSE_NUMPY=True -DUSE_RCCL=1 -DUSE_STATIC_CUDNN=0 -DUSE_STATIC_NCCL=0 -DUSE_SYSTEM_NCCL=1 /pytorch
cmake --build . --target install --config Release
##[error]Bash exited with code '1'.

```

### Versions

Used base image by running the below command from builder repo:
GPU_ARCH_TYPE=cuda GPU_ARCH_VERSION=11.8 manywheel/build_docker.sh

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu