Repository: tensorflow
Issue: tensorflow not getting cuda drivers
Labels: stat:awaiting response,type:build/install,stale,subtype: ubuntu/linux,TF 2.15

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.15.1

### Custom code

No

### OS platform and distribution

Ubuntu 22.04.4 LTS (WSL)

### Mobile device

_No response_

### Python version

3.11.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

CUDA Version: 11.4 

### GPU model and memory

_No response_

### Current behavior?

I had created the WSL setup on my Windows 11 23H2. I had installed Nvidia GPU drivers on Windows and Cuda Toolkit on my WSL (Ubuntu 22.04) as described [here](https://docs.nvidia.com/cuda/wsl-user-guide/index.html)
everythings working fine. the GPU drivers installed on Windows host are successfully got by WSL see the `# CUDA Device Query` logs in the logs section.

when i import tensorflow it say can't found any cuda libraries even though they are on the path.
```
Python 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-07-07 22:04:49.005473: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-07 22:04:49.048212: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-07 22:04:49.048305: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-07 22:04:49.049681: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-07 22:04:49.056859: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2024-07-07 22:04:49.057206: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-07 22:04:50.354998: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>>
```

# My Setup:
## Tensorflow Installation
1. Install `conda` and create a `env` and install `python=3.11`.
2. Install tensorflow from pip `tensorflow==2.15.1`

## WSL setup for enable GPU in tensorflow
1. Installed GPU drivers for the device `NVIDIA GeForce GTX 660`
2. Installed Cuda Toolkit 11.8 from [here](https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local). I installed Cuda Toolkit 12.2 and it gives me error because Cuda Toolkit 12.2 is not compatible with my GPU derivers.
checked the compatiblity from here.
    - [tensorflow comaptibility with cuda](https://www.tensorflow.org/install/source#gpu)
    - [cuda toolkit compatibility with drivers](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#id1)

_EDIT: I don't have any chance to use `tf-nightly`  or any other versions. because I am using **Tensorflow Object Detection API** and it gives lot of errors with latest tensorflow codebases like `AttributeError: module 'keras.layers' has no attribute 'experimental'` . I think the code is outdated in Object Detection API and the only solution to this error is to downgrade the tensorflow version. After spending lot of days i came with this version of tensorflow._

### Standalone code to reproduce the issue

```shell
No Code
```


### Relevant log output

```shell
$ nvidia-smi
Sun Jul  7 21:59:03 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.255      Driver Version: 475.06       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:07:00.0 N/A |                  N/A |
|ERR!    0C    P0    N/A /  N/A |    639MiB /  3072MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

$ /usr/local/cuda-11.8/extras/demo_suite/deviceQuery
/usr/local/cuda-11.8/extras/demo_suite/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA GeForce GTX 660"
  CUDA Driver Version / Runtime Version          11.4 / 11.8
  CUDA Capability Major/Minor version number:    3.0
  Total amount of global memory:                 3072 MBytes (3221225472 bytes)
  ( 5) Multiprocessors, (192) CUDA Cores/MP:     960 CUDA Cores
  GPU Max Clock rate:                            1032 MHz (1.03 GHz)
  Memory Clock rate:                             3004 Mhz
  Memory Bus Width:                              192-bit
  L2 Cache Size:                                 393216 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 7 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.8, NumDevs = 1, Device0 = NVIDIA GeForce GTX 660
Result = PASS


>>> tf.sysconfig.get_build_info()
OrderedDict([('cpu_compiler', '/usr/lib/llvm-17/bin/clang'), ('cuda_compute_capabilities', ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'compute_80']), ('cuda_version', '12.2'), ('cudnn_version', '8'), ('is_cuda_build', True), ('is_rocm_build', False), ('is_tensorrt_build', True)])
>>>
```