Repository: pytorch
Issue: Failed to build on Ubuntu 18.04 due to bad MPI linker flags
Labels: module: build,triaged

### 🐛 Describe the bug

On Ubuntu 18.04, we got an invalid MPI flags in CMakeCache.txt:
```
root@0e084f65becb:/tmp/scratch/pytorch/build# cmake --build . -- -j1
[1/2617] : && /usr/bin/g++-8 -fdebug-prefix-map='/tmp/scratch'='/usr/local/src' -g1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic -L/usr/lib/x86_64-linux-gnu/openmpi/lib;-L/usr//lib;-lmpi_cxx;-lmpi c10/benchmark/CMakeFiles/c10_intrusive_ptr_benchmark.dir/intrusive_ptr_benchmark.cpp.o -o bin/c10_intrusive_ptr_benchmark  -Wl,-rpath,/tmp/scratch/pytorch/build/lib:/usr/local/lib:  lib/libc10.so  lib/libbenchmark.a  /usr/local/lib/libglog.so.0.4.0  /usr/local/lib/libgflags.so.2.2.2  /usr/local/lib/libunwind.so  -lpthread  -pthread  /usr/lib/x86_64-linux-gnu/librt.so && :
FAILED: bin/c10_intrusive_ptr_benchmark 
: && /usr/bin/g++-8 -fdebug-prefix-map='/tmp/scratch'='/usr/local/src' -g1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic -L/usr/lib/x86_64-linux-gnu/openmpi/lib;-L/usr//lib;-lmpi_cxx;-lmpi c10/benchmark/CMakeFiles/c10_intrusive_ptr_benchmark.dir/intrusive_ptr_benchmark.cpp.o -o bin/c10_intrusive_ptr_benchmark  -Wl,-rpath,/tmp/scratch/pytorch/build/lib:/usr/local/lib:  lib/libc10.so  lib/libbenchmark.a  /usr/local/lib/libglog.so.0.4.0  /usr/local/lib/libgflags.so.2.2.2  /usr/local/lib/libunwind.so  -lpthread  -pthread  /usr/lib/x86_64-linux-gnu/librt.so && :
g++-8: fatal error: no input files
compilation terminated.
/bin/sh: 1: -L/usr//lib: not found
/bin/sh: 1: -lmpi_cxx: not found
/bin/sh: 1: -lmpi: not found
ninja: build stopped: subcommand failed.
```

CMakeCache.txt:
```
1396 //These flags will be placed after all flags passed to mpiexec.
1397 MPIEXEC_POSTFLAGS:STRING=
1398 
1399 //These flags will be directly before the executable that is being
1400 // run by mpiexec.
1401 MPIEXEC_PREFLAGS:STRING=
1402 
1403 //MPI CXX additional include directories
1404 MPI_CXX_ADDITIONAL_INCLUDE_DIRS:STRING=/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
1405 
1406 //MPI compiler for CXX
1407 MPI_CXX_COMPILER:FILEPATH=MPI_CXX_COMPILER-NOTFOUND
1408 
1409 //MPI CXX compiler wrapper include directories
1410 MPI_CXX_COMPILER_INCLUDE_DIRS:STRING=
1411 
1412 //MPI CXX compilation definitions
1413 MPI_CXX_COMPILE_DEFINITIONS:STRING=
1414 
1415 //MPI CXX compilation options
1416 MPI_CXX_COMPILE_OPTIONS:STRING=-pthread;-I/usr/lib/x86_64-linux-gnu/openmpi/include;-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
1417 
1418 //Path to a file.
1419 MPI_CXX_HEADER_DIR:PATH=/usr/lib/x86_64-linux-gnu/openmpi/include
1420 
1421 //MPI CXX include directories
1422 MPI_CXX_INCLUDE_PATH:STRING=/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
1423 
1424 //MPI CXX libraries to link against
1425 MPI_CXX_LIB_NAMES:STRING=mpi_cxx;mpi
1426 
1427 //MPI CXX linker flags
1428 MPI_CXX_LINK_FLAGS:STRING=-L/usr/lib/x86_64-linux-gnu/openmpi/lib;-L/usr//lib;-lmpi_cxx;-lmpi
1429 
1430 //If true, the MPI-2 C++ bindings are disabled using definitions.
1431 MPI_CXX_SKIP_MPICXX:BOOL=OFF
1432 
1433 //MPI C additional include directories
1434 MPI_C_ADDITIONAL_INCLUDE_DIRS:STRING=/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
1435 
1436 //MPI compiler for C
1437 MPI_C_COMPILER:FILEPATH=MPI_C_COMPILER-NOTFOUND
1438 
1439 //MPI C compiler wrapper include directories
1440 MPI_C_COMPILER_INCLUDE_DIRS:STRING=
1441 
1442 //MPI C compilation definitions
1443 MPI_C_COMPILE_DEFINITIONS:STRING=
1444 
1445 //MPI C compilation options
1446 MPI_C_COMPILE_OPTIONS:STRING=-pthread;-I/usr/lib/x86_64-linux-gnu/openmpi/include;-I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
1447 
1448 //Path to a file.
1449 MPI_C_HEADER_DIR:PATH=/usr/lib/x86_64-linux-gnu/openmpi/include
1450 
1451 //MPI C include directories
1452 MPI_C_INCLUDE_PATH:STRING=/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
1453 
1454 //MPI C libraries to link against
1455 MPI_C_LIB_NAMES:STRING=mpi
1456 
1457 //MPI C linker flags
1458 MPI_C_LINK_FLAGS:STRING=-L/usr/lib/x86_64-linux-gnu/openmpi/lib;-L/usr//lib;-lmpi
1459 
1460 //Location of the mpi library for MPI
1461 MPI_mpi_LIBRARY:FILEPATH=/usr/lib/libmpi.so
1462 
1463 //Location of the mpi_cxx library for MPI
1464 MPI_mpi_cxx_LIBRARY:FILEPATH=/usr/lib/x86_64-linux-gnu/libmpi_cxx.so
```

This is specific to the combination of Ubuntu 18.04 and CMake 3.22/3.23 so I cannot tell what's the root cause.
CMake 3.21 works fine.
CentOS 7/Debian 11 + CMake 3.23 also works fine.

When it works, `MPI_CXX_LINK_FLAGS = -pthread`.

### Versions

- Ubuntu 18.04
- gcc-8
- python 3.6
- cmake 3.23
- OpenMPI 4
- Commit 025cd69a86b4cce951c1fccb5b7160d8174e5ea9, the last one supporting stock python 3.6 on Ubuntu 18.04.

cc @malfet @seemethere