Repository: tensorflow
Issue: Tensorflow 2.15 Docker image cannot find the GPU drivers, but nvidia-smi can.
Labels: stat:awaiting tensorflower,type:bug,comp:gpu,TF 2.15

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

TF 2.15.0

### Custom code

No

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

CUDA 12.2

### GPU model and memory

NVIDIA TITAN V

### Current behavior?

Running TensorFlow 2.15.0 from within Docker image does not find GPU drivers.

nvidia-smi reports the GPUs as available.

This works as intended with a TF 2.14 image in the same machine.


### Standalone code to reproduce the issue

```shell
docker run --gpus all -it tensorflow/tensorflow:2.15.0-gpu-jupyter bash

<within the container>

# nvidia-smi

# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```


### Relevant log output

```shell
# nvidia-smi

Thu Nov 16 16:47:22 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA TITAN V                 Off | 00000000:65:00.0 Off |                  N/A |
| 29%   46C    P8              27W / 250W |   1693MiB / 12288MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2023-11-16 16:46:54.131081: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-16 16:46:54.255566: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-16 16:46:54.255623: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-16 16:46:54.276648: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-16 16:46:54.327586: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-16 16:46:54.328299: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-16 16:46:56.486851: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2256] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
```