Repository: pytorch
Issue: can not find all the deps when install the native CUDA-toolkit and torch in Redhat-like systems
Labels: module: build,module: cuda,triaged

### 🐛 Describe the bug



when I installed native cuda-toolkit then installed torch `pip install torch`,  I got `ImportError` when using `import torch`:
```
[root@af4fffdceafa data]# python3 -c "import torch;print(torch.cuda.is_available())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib64/python3.12/site-packages/torch/__init__.py", line 237, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: libcudnn.so.8: cannot open shared object 
```
the reason is:  In redhat-like system, torch installed in `/usr/local/lib64/pythonx.y/site-packages`, while its dependency installed in `/usr/local/lib/pythonx.y/site-packages`. In common case, libtorch_global_deps.so will cause `OSError` and find the dependency in a global way. However, we installed CUDA-toolkit before, which satisfied  libtorch_global_deps.so dynamic libraries  requirement:
```
[root@af4fffdceafa data]# ldd /usr/local/lib64/python3.12/site-packages/torch/lib/libtorch_global_deps.so 
     ......
        libcudart.so.12 => /usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so.12 (0x00007f3176600000)
        libnvToolsExt.so.1 => /usr/local/cuda-12.0/targets/x86_64-linux/lib/libnvToolsExt.so.1 (0x00007f3176200000)
        libgomp-a34b3233.so.1 => /usr/local/lib64/python3.12/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f3175e00000)
......
```
making the global searching does not work.

if we did not install CUDA-toolkit, the ldd command outpu is following:
```
[root@af4fffdceafa data]# ldd /usr/local/lib64/python3.12/site-packages/torch/lib/libtorch_global_deps.so 
.....
        libcudart.so.12 => not found
        libnvToolsExt.so.1 => not found
        libgomp-a34b3233.so.1 => /usr/local/lib64/python3.12/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f3165600000)
......
```
which cause OSError, and make a global searching on those so libraries.

the following scripts can reproduce it:
> installed nvidia-container-toolkit before.
```
$ docker pull fedora:latest
$ docker run -it --runtime=nvidia --gpus all fedora:latest /bin/bash
$ dnf install gcc python-pip wget which 
$ wget https://developer.download.nvidia.com/compute/cuda/12.0.0/local_installers/cuda_12.0.0_525.60.13_linux.run
# --override to avoid the check on gcc version, as the newest fedora gcc version is 13.2, which does not support cuda now
$ sh cuda_12.0.0_525.60.13_linux.run --toolkit --silent --override 
$ echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf
$ ldconfig
$ pip3 install torch numpy
$ python3 -c "import torch"
```

### Versions

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Fedora Linux 39 (Container Image) (x86_64)
GCC version: (GCC) 13.2.1 20231205 (Red Hat 13.2.1-6)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.38

Python version: 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] (64-bit runtime)
Python platform: Linux-6.1.26-2303.1.2.ocs23.x86_64-x86_64-with-glibc2.38
Is CUDA available: N/A
CUDA runtime version: 12.0.76
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 525.60.13
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
/bin/sh: line 1: lscpu: command not found

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.2.1
[conda] Could not collect

cc @malfet @seemethere @ptrblck