Repository: pytorch
Issue: Inconsistent linking flags result in error when building lib/libc10_cuda.so
Labels: module: build,module: cuda,triaged

## 🐛 Bug

Relevant portion of the output from "python setup.py build":

[346/2990] Linking CXX shared library lib/libc10_cuda.so
FAILED: lib/libc10_cuda.so
: && /opt/apps/gcc/8.3.0/bin/g++ -fPIC -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow -O3  -Wl,--no-as-needed -rdynamic -shared -Wl,-soname,libc10_cuda.so -o lib/libc10_cuda.so c10/cuda/CMakeFiles/c10_cuda.dir/CUDAStream.cpp.o c10/cuda/CMakeFiles/c10_cuda.dir/CUDACachingAllocator.cpp.o c10/cuda/CMakeFiles/c10_cuda.dir/impl/CUDAGuardImpl.cpp.o c10/cuda/CMakeFiles/c10_cuda.dir/impl/CUDATest.cpp.o  -Wl,-rpath,/export/repositories/pytorch-admix/BUILD/pytorch-cuda_1.4.0.a0_python3.8.0-1.4.0.a0/pytorch-1.4.0.a0/build/lib: -lcudart lib/libc10.so && :
/usr/bin/ld: cannot find -lcudart

The earlier output from the above  build command lists  configuration found by cmake for cuda 
as expected and as provided by the environment:
--   USE_CUDA              : 1
--     CUDA static link    : OFF
--     USE_CUDNN           : 1
--     CUDA version        : 10.1
--     cuDNN version       : 7.6.4
--     CUDA root directory : /opt/apps/cuda/10.1.243
--     CUDA library        : /opt/apps/cuda/10.1.243/lib64/stubs/libcuda.so
--     cudart library      : /opt/apps/cuda/10.1.243/lib64/libcudart.so
--     cublas library      : /opt/apps/cuda/10.1.243/lib64/libcublas.so
--     cufft library       : /opt/apps/cuda/10.1.243/lib64/libcufft.so
--     curand library      : /opt/apps/cuda/10.1.243/lib64

Build flags set:
export USE_CUDA=1 USE_CUDNN=1
export USE_MKLDNN=1 USE_MKLDNN_CBLAS=1
export USE_FFMPEG=1

Environment.
I am building from source without conda, on CentOS 7.6.
Environment is set with modules and includes  CUDA_HOME, LD_LIBRARY_PATH, CPPFLAGS, LDFLAGS which all list expected locations of installed cuda, ffmpeg, fftw, cudnn.

PyTorch version: 1.4.0a0+36d17f4
OS: CentOS Linux release 7.6.1810 (Core) 
GCC version: (GCC) 8.3.0
CMake version: version 3.12.3
Python version: 3.8
CUDA runtime version: 10.1.243
cuDNN version: /usr/lib64/libcudnn.so.7.6.4
numpy==1.17.2

After cmake configuration in  generated build.ninja file the settings for building shared library are:
# =============================================================================
# Link build statements for SHARED_LIBRARY target c10_cuda

#############################################
# Link the shared library lib/libc10_cuda.so

build lib/libc10_cuda.so: CXX_SHARED_LIBRARY_LINKER__c10_cuda c10/cuda/CMakeFiles/c10_cuda.dir/CUDAStream.cpp.o c10/cuda/CMakeFiles/c10_cuda.dir/CUDACachingAllocator.cpp.o c10/cuda/CMakeFiles/c10_cuda.dir/impl/CUDAGuardImpl.cpp.o c10/cuda/CMakeFiles/c10_cuda.dir/impl/CUDATest.cpp.o | /opt/apps/cuda/10.1.243/lib64/libcudart.so lib/libc10.so || lib/libc10.so
  LANGUAGE_COMPILE_FLAGS = -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow -O3
  LINK_FLAGS = -Wl,--no-as-needed -rdynamic
  LINK_LIBRARIES = -Wl,-rpath,/export/repositories/pytorch-admix/BUILD/pytorch-cuda_1.4.0.a0_python3.8.0-1.4.0.a0/pytorch-1.4.0.a0/build/lib: -lcudart lib/libc10.so
  OBJECT_DIR = c10/cuda/CMakeFiles/c10_cuda.dir
  POST_BUILD = :
  PRE_LINK = :
  SONAME = libc10_cuda.so
  SONAME_FLAG = -Wl,-soname,
  TARGET_FILE = lib/libc10_cuda.so

It looks like while libcudart.so is found by cmake configure process, its location is not passed 
correctly during the compile stage and per this build.ninja snippet the compilation fails. 

My LDFLAGS  variable is set to include cuda and compiler  libraries:

LDFLAGS=-L/opt/apps/cuda/10.1.243/lib64 -Wl,-rpath,/opt/apps/cuda/10.1.243/lib64 -L/opt/apps/cuda/10.1.243/nvvm/lib64 -Wl,-rpath,/opt/apps/cuda/10.1.243/nvvm/lib64 -L/opt/apps/cuda/10.1.243/cublas/lib64 -Wl,-rpath,/opt/apps/cuda/10.1.243/cublas/lib64 -L/opt/apps/cuda/10.1.243/extras/CUPTI/lib64/lib64 -Wl,-rpath,/opt/apps/cuda/10.1.243/extras/CUPTI/lib64/lib64 -L/opt/apps/cuda/10.1.243/extras/Debugger/lib64/lib64 -Wl,-rpath,/opt/apps/cuda/10.1.243/extras/Debugger/lib64/lib64 -L/opt/apps/fftw/3.3.8/lib -Wl,-rpath,/opt/apps/fftw/3.3.8/lib -L/opt/intel/compilers_and_libraries_2019.5.281/linux/tbb/lib/intel64 -L/opt/intel/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64 -L/opt/intel/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64 -L/opt/apps/gcc/8.3.0/lib64 -Wl,-rpath,/opt/apps/gcc/8.3.0/lib64

But it looks like LDFLAGS  variable is not taken into account. Both LD_LIBRARY_PATH and LIBRARY_PATH are set to include cuda libraries as well. 
Per compile from source docs or setup.py, I don't see any other variables that need to be set.

Please advise what changes are needed  to  compile pytorch with cuda from source.
I have cpu only pytorch version compiled successfully using the same settings (minus cuda).


cc @ngimel