Repository: pytorch
Issue: PyTorch / libtorch executables fail when built against libcuda stub library
Labels: high priority,module: binaries,module: build,triaged

## 🐛 Bug

PyTorch's cmake setup for CUDA includes a couple `stubs/` directories in the search path for `libcuda.so`:

https://github.com/pytorch/pytorch/blob/v1.4.0/cmake/public/cuda.cmake#L179

Searching for libcuda in the stubs/ directories is needed in various cases, for example when:

- the build host is GPU-less and has CUDA Toolkit but not the GPU driver installed, or
- the build toolchain doesn't include normal system search paths that would normally house libcuda. This is likely if the Anaconda toolchain is being used to build PyTorch.

However, PyTorch's build setup tells cmake to add any non-PyTorch-local libraries to built objects' `RPATH`:

https://github.com/pytorch/pytorch/blob/v1.4.0/cmake/Dependencies.cmake#L14


So the few objects that link directly to libcuda (e.g. `lib/libcaffe2_nvrtc.so` and `bin/test_dist_autograd`) may end up with the libcuda stubs/ directory included in RPATH.

During loading, RPATH has priority over normal system search paths and even `LD_LIBRARY_PATH`. So these objects will prefer the libcuda stub, even on hosts where libcuda is present in the normal location.

The same cmake setup is re-used for libtorch, so the same will occur for libtorch apps as well:

https://discuss.pytorch.org/t/torch-is-available-returns-false/73753


The libcuda stub is not a functional CUDA runtime, so PyTorch and libtorch objects affected by this won't be able to use CUDA even if libcuda is installed in the normal system search path.


This change would prevent a libcuda stubs directory from being added to objects' RPATH, but doesn't seem like a great solution. It abuses a cmake internal mechanism to trick it into avoiding adding the stubs directory.

```
diff --git a/cmake/public/cuda.cmake b/cmake/public/cuda.cmake
index a5c50b90df..3a89c9e8ad 100644
--- a/cmake/public/cuda.cmake
+++ b/cmake/public/cuda.cmake
@@ -187,6 +187,20 @@ find_library(CUDA_NVRTC_LIB nvrtc
     PATHS ${CUDA_TOOLKIT_ROOT_DIR}
     PATH_SUFFIXES lib lib64 lib/x64)

+# Configuration elsewhere (CMAKE_INSTALL_RPATH_USE_LINK_PATH) arranges that any
+# library used during linking will be added to the objects' RPATHs. That's
+# never correct for a libcuda stub library (the stub library is suitable for
+# linking, but not for runtime, and if present in RPATH will be prioritized
+# ahead of system search path). If libcuda was found in the "stubs" directory,
+# abuse CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES to instruct cmake NOT to
+# include it in RPATH This leaves us without a known path to libcuda in the
+# objects, but there's no help for that if the only libcuda visible at build
+# time is the stub.
+if(CUDA_CUDA_LIB MATCHES "stubs")
+  get_filename_component(LIBCUDA_STUB_DIR ${CUDA_CUDA_LIB} DIRECTORY)
+  list(APPEND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES ${LIBCUDA_STUB_DIR})
+endif()
+
 # Create new style imported libraries.
 # Several of these libraries have a hardcoded path if CAFFE2_STATIC_LINK_CUDA
 # is set. This path is where sane CUDA installations have their static
```

I hope someone more familiar with cmake and the general pytorch build setup might have a better idea.


## To Reproduce

Steps to reproduce the behavior:

1. Built pytorch on a system with CUDA Toolkit but not GPU driver installed
1. `objdump -p <install_dir>/lib/libcaffe2_nvrtc.so | grep RPATH`

## Expected behavior

PyTorch and libtorch objects should build successfully against libcuda stub, and then run successfully against "real" libcuda (at least when libcuda is present in a default system search path).

## Environment

 - PyTorch Version (e.g., 1.0): 1.4.0
 - OS (e.g., Linux): Linux
 - How you installed PyTorch (`conda`, `pip`, source): source
 - Build command you used (if compiling from source): `python setup.py install`
 - Python version: `Python 3.6.9 :: Anaconda, Inc.`
 - CUDA/cuDNN version: 10.2 / 7.6.5
 - GPU models and configuration: V100
 - Any other relevant information: Using Anaconda toolchain to build

cc @ezyang @gchanan @zou3519 @seemethere