Repository: pytorch
Issue: caffe does not respect CUDNN_LIB_DIR when building from source (cmake)
Labels: module: build,triaged,module: nvfuser

### 🐛 Describe the bug

Per the [setup.py](https://github.com/pytorch/pytorch/blob/main/setup.py#L171) and past experience from torch 1.13, setting the `CUDNN_LIB_DIR` environment variable should point the whole pytorch build to the CUDNN headers and dll's during custom builds. For good measure, I set all of these:
```
export CUDNN_LIB_DIR="<some local path>"
export CUDNN_INCLUDE_DIR="$CUDNN_LIB_DIR/include"
export CUDNN_INCLUDE_PATH="$CUDNN_INCLUDE_DIR"
export CUDNN_LIBRARY="$CUDNN_LIB_DIR/lib/libcudnn.so"
export CUDNN_LIBRARY_PATH="$CUDNN_LIBRARY"
```

However, I'm now trying to build with branch `release/2.0` and getting these errors:

```
[ 98%] Building CXX object third_party/nvfuser/CMakeFiles/nvfuser_codegen.dir/csrc/lower_index_hoist.cpp.o 
In file included from /.../libtorch/pytorch/caffe2/core/context_gpu.h:20, 
                 from /.../libtorch/pytorch/modules/detectron/sample_as_op.cu:23: 
/.../libtorch/pytorch/caffe2/core/common_cudnn.h:16:10: fatal error: cudnn.h: No such file or directory 
 #include <cudnn.h> 
          ^~~~~~~~~ 
compilation terminated. 
gmake[2]: *** [modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/build.make:272: modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/sample_as_op.cu.o] Error 1 
gmake[2]: *** Waiting for unfinished jobs.... 
[ 98%] Building CXX object third_party/nvfuser/CMakeFiles/nvfuser_codegen.dir/csrc/lower_insert_syncs.cpp.o 
In file included from /.../libtorch/pytorch/caffe2/core/context_gpu.h:20, 
                 from /.../libtorch/pytorch/modules/detectron/select_smooth_l1_loss_op.cu:17: 
/.../libtorch/pytorch/caffe2/core/common_cudnn.h:16:10: fatal error: cudnn.h: No such file or directory 
 #include <cudnn.h> 
          ^~~~~~~~~ 
compilation terminated. 
gmake[2]: *** [modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/build.make:286: modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/select_smooth_l1_loss_op.cu.o] Error 1 
In file included from /.../libtorch/pytorch/caffe2/core/context_gpu.h:20, 
                 from /.../libtorch/pytorch/modules/detectron/upsample_nearest_op.cu:60: 
/.../libtorch/pytorch/caffe2/core/common_cudnn.h:16:10: fatal error: cudnn.h: No such file or directory 
 #include <cudnn.h> 
          ^~~~~~~~~ 
compilation terminated. 
[ 98%] Building CXX object third_party/nvfuser/CMakeFiles/nvfuser_codegen.dir/csrc/lower_instrument.cpp.o 
gmake[2]: *** [modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/build.make:370: modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/upsample_nearest_op.cu.o] Error 1 
In file included from /.../libtorch/pytorch/caffe2/core/context_gpu.h:20, 
                 from /.../libtorch/pytorch/modules/detectron/smooth_l1_loss_op.cu:17: 
/.../libtorch/pytorch/caffe2/core/common_cudnn.h:16:10: fatal error: cudnn.h: No such file or directory 
 #include <cudnn.h> 
          ^~~~~~~~~ 
compilation terminated. 
gmake[2]: *** [modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/build.make:328: modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/smooth_l1_loss_op.cu.o] Error 1
... (several more of these)
```

My cmake script looks like this:
```
# ... export environment variables

conda install -y \
  astunparse \
  numpy \
  ninja \
  pyyaml \
  setuptools \
  cmake \
  cffi \
  typing_extensions \
  future \
  six \
  requests \
  dataclasses \
  glog==0.4.0 \
  mkl \
  mkl-include

conda install -y -c pytorch magma-cuda$MAGMA_VERSION

git clone -b "$TORCH_BRANCH" https://github.com/pytorch/pytorch.git
cd pytorch
git submodule sync
git submodule update --init --recursive

mkdir build
cd build

cmake <a lot of flags> ..
cmake --build . --target install --config Release -- -j $(nproc)
```

I have a hacky workaround by setting
```
export CPATH="$CPATH:$CUDNN_INCLUDE_DIR"
```
but I'm guessing there could be an easy fix in caffe to respect the CUDNN environment variable(s).

### Versions

CUDA 12.1
CUDNN 8.8
(lib)torch 2.0

cc @malfet @seemethere @kevinstephano @jjsjann123