Repository: pytorch
Issue: nvcc -broken when use Libtorch 
Labels: needs reproduction,module: windows,module: cpp,triaged

I want to use libtorch in windows10, I met the following problem during cmake.


build# cmake ..
-- Building for: MinGW Makefiles
-- The C compiler identification is GNU 8.1.0
-- The CXX compiler identification is GNU 8.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: D:/Qt/Tools/mingw810_64/bin/gcc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: D:/Qt/Tools/mingw810_64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8 (found version "11.8")
-- The CUDA compiler identification is NVIDIA 11.8.89
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - failed
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin/nvcc.exe
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin/nvcc.exe - broken
CMake Error at D:/Qt/Tools/CMake_64/share/cmake-3.27/Modules/CMakeTestCUDACompiler.cmake:100 (message):
  The CUDA compiler

    "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin/nvcc.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: 'D:/pytorch/cpp/demo/build/CMakeFiles/CMakeScratch/TryCompile-qq8on1'
    
    Run Build Command(s): D:/Qt/Tools/CMake_64/bin/cmake.exe -E env VERBOSE=1 D:/Qt/Tools/mingw810_64/bin/mingw32-make.exe -f Makefile cmTC_fd7e9/fast
    D:/Qt/Tools/mingw810_64/bin/mingw32-make.exe  -f CMakeFiles\cmTC_fd7e9.dir\build.make CMakeFiles/cmTC_fd7e9.dir/build
    mingw32-make.exe[1]: Entering directory 'D:/pytorch/cpp/demo/build/CMakeFiles/CMakeScratch/TryCompile-qq8on1'
    Building CUDA object CMakeFiles/cmTC_fd7e9.dir/main.cu.obj
    C:\PROGRA~1\NVIDIA~2\CUDA\v11.8\bin\nvcc.exe -forward-unknown-to-host-compiler   -D_WINDOWS -Xcompiler="/W3 /GR /EHsc"  -Xcompiler="-MDd  -Zi -Ob0 -Od /RTC1" -MD -MT CMakeFiles/cmTC_fd7e9.dir/main.cu.obj -MF CMakeFiles\cmTC_fd7e9.dir\main.cu.obj.d -x cu -c D:\pytorch\cpp\demo\build\CMakeFiles\CMakeScratch\TryCompile-qq8on1\main.cu -o CMakeFiles\cmTC_fd7e9.dir\main.cu.obj -Xcompiler=-FdCMakeFiles\cmTC_fd7e9.dir/,-FS      
    main.cu
    Linking CUDA executable cmTC_fd7e9.exe
    D:\Qt\Tools\CMake_64\bin\cmake.exe -E cmake_link_script CMakeFiles\cmTC_fd7e9.dir\link.txt --verbose=1
    D:\Qt\Tools\CMake_64\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_fd7e9.dir --rc=D:\Qt\Tools\mingw810_64\bin\windres.exe --mt="" --manifests -- D:\Qt\Tools\mingw810_64\bin\ld.exe /nologo @CMakeFiles\cmTC_fd7e9.dir\objects1.rsp  /out:cmTC_fd7e9.exe /implib:cmTC_fd7e9.lib /pdb:D:\pytorch\cpp\demo\build\CMakeFiles\CMakeScratch\TryCompile-qq8on1\cmTC_fd7e9.pdb /version:0.0 /debug /INCREMENTAL @CMakeFiles\cmTC_fd7e9.dir\linkLibs.rsp -LIBPATH:"C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/lib/x64"
    Visual Studio Incremental Link with embedded manifests
    Create CMakeFiles\cmTC_fd7e9.dir/manifest.rc
    Create empty: CMakeFiles\cmTC_fd7e9.dir/embed.manifest
    RC Pass 1:
    D:\Qt\Tools\mingw810_64\bin\windres.exe /fo CMakeFiles\cmTC_fd7e9.dir/manifest.res CMakeFiles\cmTC_fd7e9.dir/manifest.rc
    RC Pass 1: command "D:\Qt\Tools\mingw810_64\bin\windres.exe /fo CMakeFiles\cmTC_fd7e9.dir/manifest.res CMakeFiles\cmTC_fd7e9.dir/manifest.rc" failed (exit code 1) with the following output:
    Usage: D:\Qt\Tools\mingw810_64\bin\windres.exe [option(s)] [input-file] [output-file]
     The options are:
      -i --input=<file>            Name input file
      -o --output=<file>           Name output file
      -J --input-format=<format>   Specify input format
      -O --output-format=<format>  Specify output format
      -F --target=<target>         Specify COFF target
         --preprocessor=<program>  Program to use to preprocess rc file
         --preprocessor-arg=<arg>  Additional preprocessor argument
      -I --include-dir=<dir>       Include directory when preprocessing rc file
      -D --define <sym>[=<val>]    Define SYM when preprocessing rc file
      -U --undefine <sym>          Undefine SYM when preprocessing rc file
      -v --verbose                 Verbose - tells you what it's doing
      -c --codepage=<codepage>     Specify default codepage
      -l --language=<val>          Set language when reading rc file
         --use-temp-file           Use a temporary file instead of popen to read
                                   the preprocessor output
         --no-use-temp-file        Use popen (default)
      -r                           Ignored for compatibility with rc
      @<file>                      Read options from <file>
      -h --help                    Print this help message
      -V --version                 Print version information
    FORMAT is one of rc, res, or coff, and is deduced from the file name
    extension if not specified.  A single file name is an input file.
    No input-file is stdin, default rc.  No output-file is stdout, default rc.
    D:\Qt\Tools\mingw810_64\bin\windres.exe: supported targets: pe-x86-64 pei-x86-64 pe-bigobj-x86-64 elf64-x86-64 elf64-l1om elf64-k1om pe-i386 pei-i386 elf32-i386 elf32-iamcu elf64-little elf64-big elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex
    mingw32-make.exe[1]: *** [CMakeFiles\cmTC_fd7e9.dir\build.make:101: cmTC_fd7e9.exe] Error -1
    mingw32-make.exe[1]: Leaving directory 'D:/pytorch/cpp/demo/build/CMakeFiles/CMakeScratch/TryCompile-qq8on1'
    mingw32-make.exe: *** [Makefile:126: cmTC_fd7e9/fast] Error 2





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  D:/GoogleDownload/libtorch-win-shared-with-deps-2.3.1+cu118/libtorch/share/cmake/Caffe2/public/cuda.cmake:47 (enable_language)
  D:/GoogleDownload/libtorch-win-shared-with-deps-2.3.1+cu118/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
  D:/GoogleDownload/libtorch-win-shared-with-deps-2.3.1+cu118/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:5 (find_package)


-- Configuring incomplete, errors occurred!

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jbschlosser