Repository: pytorch
Issue: Unable to compile PyTorch when libcudart_static.so is not available 
Labels: module: build,module: cuda,triaged

### 🐛 Describe the bug

I used to build PyTorch from source on my Fedora (35) devbox and never add issue so far.

Upgrading to CUDA 11.5 (_negativo repository_) somehow breaks the compilation process with the following: 

I saw some older issue related to static linking with cudart when `USE_DISTRIBUTED=ON` was fixing the issue:
-> https://github.com/pytorch/pytorch/issues/14805 (_That was already me posting ... 🙄, maybe I'm the issue 😁_)

I tried both `USE_DISTRIBUTED=ON/OFF` the error is the same in this case, same error.

Thanks for the awesome work! 🤗

`CUDAHOSTCXX=/usr/bin/cuda-g++ CC=cuda-gcc CXX=cuda-g++ CFLAGS="-DTHRUST_IGNORE_CUB_VERSION_CHECK" TORCH_CUDA_ARCH_LIST="8.6+PTX" CMAKE_GENERATOR="Ninja" CUDNN_LIB_DIR=/usr/lib64 BUILD_CAFFE2_OPS=OFF BUILD_CAFFE2=OFF USE_FBGEMM=OFF USE_OPENCV=OFF BUILD_TEST=OFF USE_KINETO=ON USE_MKLDNN=ON BLAS=MKL ATEN_THREADING=OMP USE_FFMPEG=OFF USE_LEVELDB=OFF USE_REDIS=OFF USE_NNPACK=ON USE_QNNPACK=ON USE_NCCL=OFF USE_DISTRIBUTED=OFF CUDA_HOME=/usr/local/cuda USE_SYSTEM_LIBS=ON CUDA_USE_STATIC_CUDA_RUNTIME=OFF python setup.py bdist_wheel`

```bash
Building wheel torch-1.11.0a0+git78bea1b
-- Building version 1.11.0a0+git78bea1b
cmake3 -GNinja -DATEN_THREADING=OMP -DBLAS=MKL -DBUILD_CAFFE2=OFF -DBUILD_CAFFE2_OPS=OFF -DBUILD_PYTHON=True -DBUILD_TEST=False -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR=Ninja -DCMAKE_INSTALL_PREFIX=/speedy/Workspace/pytorch/torch -DCMAKE_PREFIX_PATH=/data/Workspace/miniconda/envs/default/lib/python3.9/site-packages -DCUDA_HOST_COMPILER=/usr/bin/cuda-g++ -DCUDNN_LIBRARY=/usr/lib64 -DNUMPY_INCLUDE_DIR=/home/mfuntowicz/.local/lib/python3.9/site-packages/numpy-1.20.0rc2-py3.9-linux-x86_64.egg/numpy/core/include -DPYTHON_EXECUTABLE=/data/Workspace/miniconda/envs/default/bin/python -DPYTHON_INCLUDE_DIR=/data/Workspace/miniconda/envs/default/include/python3.9 -DPYTHON_LIBRARY=/data/Workspace/miniconda/envs/default/lib/libpython3.9.a -DTORCH_BUILD_VERSION=1.11.0a0+git78bea1b -DUSE_DISTRIBUTED=OFF -DUSE_FBGEMM=OFF -DUSE_FFMPEG=OFF -DUSE_KINETO=ON -DUSE_LEVELDB=OFF -DUSE_MKLDNN=ON -DUSE_NCCL=OFF -DUSE_NNPACK=ON -DUSE_NUMPY=True -DUSE_OPENCV=OFF -DUSE_QNNPACK=ON -DUSE_REDIS=OFF -DUSE_SYSTEM_LIBS=ON /speedy/Workspace/pytorch
-- The CXX compiler identification is GNU 9.3.0
-- The C compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/cuda-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/bin/cuda-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning (dev) at /usr/share/cmake/Modules/CMakeDependentOption.cmake:84 (message):
  Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
  Syntax.  Run "cmake --help-policy CMP0127" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
  CMakeLists.txt:253 (cmake_dependent_option)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Performing Test COMPILER_WORKS
-- Performing Test COMPILER_WORKS - Success
-- Performing Test SUPPORT_GLIBCXX_USE_C99
-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed
-- Turning off deprecation warning due to glog.
-- Performing Test C_HAS_AVX_1
-- Performing Test C_HAS_AVX_1 - Failed
-- Performing Test C_HAS_AVX_2
-- Performing Test C_HAS_AVX_2 - Success
-- Performing Test C_HAS_AVX2_1
-- Performing Test C_HAS_AVX2_1 - Failed
-- Performing Test C_HAS_AVX2_2
-- Performing Test C_HAS_AVX2_2 - Success
-- Performing Test C_HAS_AVX512_1
-- Performing Test C_HAS_AVX512_1 - Failed
-- Performing Test C_HAS_AVX512_2
-- Performing Test C_HAS_AVX512_2 - Success
-- Performing Test CXX_HAS_AVX_1
-- Performing Test CXX_HAS_AVX_1 - Failed
-- Performing Test CXX_HAS_AVX_2
-- Performing Test CXX_HAS_AVX_2 - Success
-- Performing Test CXX_HAS_AVX2_1
-- Performing Test CXX_HAS_AVX2_1 - Failed
-- Performing Test CXX_HAS_AVX2_2
-- Performing Test CXX_HAS_AVX2_2 - Success
-- Performing Test CXX_HAS_AVX512_1
-- Performing Test CXX_HAS_AVX512_1 - Failed
-- Performing Test CXX_HAS_AVX512_2
-- Performing Test CXX_HAS_AVX512_2 - Success
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
-- Found CUDA: /usr/local/cuda (found version "11.5")
CMake Error at /usr/share/cmake/Modules/CMakeDetermineCompilerId.cmake:726 (message):
  Compiling the CUDA compiler identification source file
  "CMakeCUDACompilerId.cu" failed.

  Compiler: /usr/bin/nvcc

  Build flags:

  Id flags: --keep;--keep-dir;tmp;-ccbin=/usr/bin/cuda-g++ -v



  The output was:

  1

  #$ _NVVM_BRANCH_=nvvm

  #$ _SPACE_=

  #$ _CUDART_=cudart

  #$ _HERE_=/usr/bin

  #$ _THERE_=/usr/bin

  #$ _TARGET_SIZE_=

  #$ _TARGET_DIR_=

  #$ _TARGET_SIZE_=64

  #$ NVVMIR_LIBRARY_DIR=/usr/share/cuda

  #$
  PATH=/usr/libexec/cuda/open64/bin:/data/Workspace/miniconda/envs/default/bin:/data/Workspace/miniconda/condabin:/home/mfuntowicz/.cargo/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/mfuntowicz/bin:/var/lib/snapd/snap/bin


  #$ INCLUDES="-I/usr/include/cuda"

  #$ LIBRARIES="-L/usr/lib64"

  #$ CUDAFE_FLAGS=

  #$ PTXAS_FLAGS=

  #$ rm tmp/a_dlink.reg.c

  #$ "/usr/bin"/cuda-g++ -D__CUDA_ARCH__=520 -D__CUDA_ARCH_LIST__=520 -E -x
  c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__
  "-I/usr/include/cuda" -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=5
  -D__CUDACC_VER_BUILD__=50 -D__CUDA_API_VER_MAJOR__=11
  -D__CUDA_API_VER_MINOR__=5 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
  "cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
  "tmp/CMakeCUDACompilerId.cpp1.ii"

  #$ cicc --c++14 --gnu_version=90300 --display_error_number
  --orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
  "/speedy/Workspace/pytorch/build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu"
  --allow_managed -arch compute_52 -m64 --no-version-ident -ftz=0 -prec_div=1
  -prec_sqrt=1 -fmad=1 --include_file_name "CMakeCUDACompilerId.fatbin.c"
  -tused --gen_module_id_file --module_id_file_name
  "tmp/CMakeCUDACompilerId.module_id" --gen_c_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.c" --stub_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.stub.c" --gen_device_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.gpu" "tmp/CMakeCUDACompilerId.cpp1.ii" -o
  "tmp/CMakeCUDACompilerId.ptx"

  #$ ptxas -arch=sm_52 -m64 "tmp/CMakeCUDACompilerId.ptx" -o
  "tmp/CMakeCUDACompilerId.sm_52.cubin"

  #$ fatbinary --create="tmp/CMakeCUDACompilerId.fatbin" -64
  --cicc-cmdline="-ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 "
  "--image3=kind=elf,sm=52,file=tmp/CMakeCUDACompilerId.sm_52.cubin"
  "--image3=kind=ptx,sm=52,file=tmp/CMakeCUDACompilerId.ptx"
  --embedded-fatbin="tmp/CMakeCUDACompilerId.fatbin.c"

  #$ "/usr/bin"/cuda-g++ -D__CUDA_ARCH_LIST__=520 -E -x c++ -D__CUDACC__
  -D__NVCC__ "-I/usr/include/cuda" -D__CUDACC_VER_MAJOR__=11
  -D__CUDACC_VER_MINOR__=5 -D__CUDACC_VER_BUILD__=50
  -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=5
  -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64
  "CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.cpp4.ii"

  #$ cudafe++ --c++14 --gnu_version=90300 --display_error_number
  --orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
  "/speedy/Workspace/pytorch/build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu"
  --allow_managed --m64 --parse_templates --gen_c_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
  "CMakeCUDACompilerId.cudafe1.stub.c" --module_id_file_name
  "tmp/CMakeCUDACompilerId.module_id" "tmp/CMakeCUDACompilerId.cpp4.ii"

  #$ "/usr/bin"/cuda-g++ -D__CUDA_ARCH__=520 -D__CUDA_ARCH_LIST__=520 -c -x
  c++ -DCUDA_DOUBLE_MATH_FUNCTIONS "-I/usr/include/cuda" -m64
  "tmp/CMakeCUDACompilerId.cudafe1.cpp" -o "tmp/CMakeCUDACompilerId.o"

  #$ nvlink -m64 --arch=sm_52 --register-link-binaries="tmp/a_dlink.reg.c"
  "-L/usr/lib64" -cpu-arch=X86_64 "tmp/CMakeCUDACompilerId.o" -lcudadevrt -o
  "tmp/a_dlink.sm_52.cubin"

  #$ fatbinary --create="tmp/a_dlink.fatbin" -64 --cicc-cmdline="-ftz=0
  -prec_div=1 -prec_sqrt=1 -fmad=1 " -link
  "--image3=kind=elf,sm=52,file=tmp/a_dlink.sm_52.cubin"
  --embedded-fatbin="tmp/a_dlink.fatbin.c"

  #$ "/usr/bin"/cuda-g++ -D__CUDA_ARCH_LIST__=520 -c -x c++
  -DFATBINFILE="\"tmp/a_dlink.fatbin.c\""
  -DREGISTERLINKBINARYFILE="\"tmp/a_dlink.reg.c\"" -I.
  -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION=
  -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__ "-I/usr/include/cuda"
  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=5
  -D__CUDACC_VER_BUILD__=50 -D__CUDA_API_VER_MAJOR__=11
  -D__CUDA_API_VER_MINOR__=5 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -m64
  "/usr/bin/crt/link.stub" -o "tmp/a_dlink.o"

  #$ "/usr/bin"/cuda-g++ -D__CUDA_ARCH_LIST__=520 -m64 -Wl,--start-group
  "tmp/a_dlink.o" "tmp/CMakeCUDACompilerId.o" "-L/usr/lib64" -lcudadevrt
  -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -o "a.out"

  /usr/bin/ld: cannot find -lcudart_static

  collect2: error: ld returned 1 exit status

  # --error 0x1 --
```

### Versions

```bash
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Fedora Linux 35 (Thirty Five) (x86_64)
GCC version: (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7)
Clang version: 13.0.0 (Fedora 13.0.0-3.fc35)
CMake version: version 3.22.1
Libc version: glibc-2.34

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.10-200.fc35.x86_64-x86_64-with-glibc2.34
Is CUDA available: N/A
CUDA runtime version: 11.5.50
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 495.46
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.2.4
/usr/lib64/libcudnn_adv_infer.so.8.2.4
/usr/lib64/libcudnn_adv_train.so.8.2.4
/usr/lib64/libcudnn_cnn_infer.so.8.2.4
/usr/lib64/libcudnn_cnn_train.so.8.2.4
/usr/lib64/libcudnn_ops_infer.so.8.2.4
/usr/lib64/libcudnn_ops_train.so.8.2.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.0rc2
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0a0+8a347b6
[pip3] torchmetrics==0.4.1
[conda] blas                      1.0                         mkl
[conda] magma-cuda115             2.5.2                         1    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-devel                 2021.4.0           h66538d2_640
[conda] mkl-include               2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.21.2           py39h20f2e39_0
[conda] numpy-base                1.21.2           py39h79a1101_0
[conda] torch                     1.10.0a0+gitd6d286f          pypi_0    pypi
[conda] torchaudio                0.10.0a0+8a347b6          pypi_0    pypi
[conda] torchmetrics              0.4.1                    pypi_0    pypi
```

cc @malfet @seemethere @ngimel