Repository: tensorflow
Issue: Cannot dlopen some GPU libraries [can't find cuda driver] in rhel9
Labels: stat:awaiting tensorflower,type:bug,type:build/install,2.17

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

v2.17.0-rc1-2-gad6d8cc177d 2.17.0

### Custom code

Yes

### OS platform and distribution

Redhat enterprise 9.4 base image

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

12.2

### GPU model and memory

Tesla T4, 15360MiB

### Current behavior?

What is your question?

Describe the bug

Error when I run GPU test, wondering if my docker linux kernel version is too low? The reason I am asking is that, my ubuntu22 is working fine.

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723846435.253301 203 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-08-16 22:13:55.253747: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2343] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
"
My Kernel version,

bash-5.1# uname -a
Linux mlops-test-failed 5.10.220-209.869.amzn2.x86_64 https://github.com/rapidsai/cudf/issues/1 SMP Wed Jul 17 15:10:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

My OS/Docker image distro,
"
bash-5.1# uname -m && cat /etc/*release
x86_64
NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
Red Hat Enterprise Linux release 9.4 (Plow)
Red Hat Enterprise Linux release 9.4 (Plow)
"
Both my ubuntu 22 and rhel9 were showing the nvidia-smi ok like the following.

bash-5.1# nvidia-smi
Fri Aug 16 22:29:43 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.06 Driver Version: 535.183.06 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 |
| N/A 33C P8 11W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

I am checking the following url,
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

and it says the rhel9 kernel version has to be 5.14.0-427.

Your input is appreciated!

### Standalone code to reproduce the issue

```shell
def run_gpu_test():
    gpus = tf.config.list_logical_devices('GPU')
    print("Num GPUs Available: ", len(gpus))

run_gpu_test()

In Ubuntu22, it prints gpu number 1 with all the gpu information and in rhel9 it does not.

This is a Pod we created in eks, and by exec to the pod, we pasted the debugging information in the above section. I did nvidia-smi and both ubuntu22 and rhel9 shows GPU fine. Ubuntu22 works fine but not rhel9. The node we created is a AWS G4 instance, so it has tensorflow 12.7 and cuda 12.2 and we installed nvidia-plugin. I think this should be very easy to reproduce not sure if the aws kernel version matters.
```


### Relevant log output

```shell
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723846435.253301 203 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-08-16 22:13:55.253747: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2343] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
"
```