Repository: tensorflow
Issue: TF-TRT Warning: Could not find TensorRT
Labels: stat:awaiting tensorflower,type:build/install,comp:gpu,subtype: ubuntu/linux,comp:gpu:tensorrt,TF 2.15

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.15.0

### Custom code

No

### OS platform and distribution

Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

11.2

### CUDA/cuDNN version

12.4

### GPU model and memory

RTX 3050 TI

### Current behavior?

Hey everybody,

I have tried installing tensorflow from my ubuntu 22.04 terminal with anaconda environment.  I am getting Can not find TensorRT.  I have a 4 monitor setup with Ubuntu and the 550 NVIDIA driver for the RTX 3050 TI does not work on my machine.  I have to run the 535 driver, and I think 12.4 CUDA is supported.  It installed the 550 driver automatically and I revert back to the 535 driver.  But after all that I still ended up with the error below.

I have tried uninstalling, reinstalling, and can't get it to work.  I am a graduate student taking a Machine Learning course.  I am spending more time debugging this issue then actually coding.  

Could somebody help me resolve?

### Standalone code to reproduce the issue

```shell
+---------------------------------------------------------------------------------------+                                                                                                                          
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |                                                                                                                          
|-----------------------------------------+----------------------+----------------------+                                                                                                                          
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |                                                                                                                          
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |                                                                                                                          
|                                         |                      |               MIG M. |                                                                                                                          
|=========================================+======================+======================|                                                                                                                          
|   0  NVIDIA GeForce RTX 3050 ...    Off | 00000000:01:00.0  On |                  N/A |                                                                                                                          
| N/A   66C    P8               8W /  35W |   1011MiB /  4096MiB |      0%      Default |                                                                                                                          
|                                         |                      |                  N/A |                                                                                                                          
+-----------------------------------------+----------------------+----------------------+                                                                                                                          
                                                                                                                                                                                                                   
+---------------------------------------------------------------------------------------+                                                                                                                          
| Processes:                                                                            |                                                                                                                          
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |                                                                                                                          
|        ID   ID                                                             Usage      |                                                                                                                          
|=======================================================================================|                                                                                                                          
|    0   N/A  N/A      2812      G   /usr/lib/xorg/Xorg                          648MiB |                                                                                                                          
|    0   N/A  N/A      3023      G   /usr/bin/gnome-shell                        148MiB |                                                                                                                          
|    0   N/A  N/A     78082      G   ...az/anaconda3/envs/skynet/bin/python        1MiB |                                                                                                                          
|    0   N/A  N/A     78475      G   ...seed-version=20240329-165146.919000       96MiB |                                                                                                                          
|    0   N/A  N/A     79970      G   ...gnu/webkit2gtk-4.0/WebKitWebProcess       68MiB |                                                                                                                          
+---------------------------------------------------------------------------------------+   
(base) thenaz@Skynet:~/Downloads$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 2291 (rev a1)
(base) thenaz@Skynet:~/Downloads$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver                                                                                                                                                                              
Copyright (c) 2005-2024 NVIDIA Corporation                                                                                                                                                                         
Built on Tue_Feb_27_16:19:38_PST_2024                                                                                                                                                                              
Cuda compilation tools, release 12.4, V12.4.99                                                                                                                                                                     
Build cuda_12.4.r12.4/compiler.33961263_0
```


### Relevant log output

```shell
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"                                                                                  
2024-03-30 14:29:58.668656: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.                                                                                                                                 
2024-03-30 14:29:58.691384: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 
2024-03-30 14:29:58.691405: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered  
2024-03-30 14:29:58.692006: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered                                                                                                                                                                                                                 
2024-03-30 14:29:58.695703: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.                         
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-30 14:29:59.119119: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-30 14:29:59.660586: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-30 14:29:59.675221: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-30 14:29:59.675340: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
```