Repository: tensorflow
Issue: CUDA Messages in TensorFlow 2.17.0 and tf-nightly
Labels: stat:awaiting tensorflower,type:build/install,comp:gpu,2.17

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.17.0

### Custom code

Yes

### OS platform and distribution

Linux

### Mobile device

_No response_

### Python version

3.12

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Hi,

I've updated TensorFlow to version 2.17.0 and noticed that several CUDA-related messages appear during model training.

Example messages:
```
I0000 00:00:1720821406.093527 2748382 cuda_executor.cc:821] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720821406.127659 2748382 cuda_executor.cc:821] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720821406.127833 2748382 cuda_executor.cc:821] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720821406.161631 2748382 cuda_executor.cc:821] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
```

```
I0000 00:00:1720821413.362793 2748566 cuda_dnn.cc:530] Loaded cuDNN version 8907
W0000 00:00:1720821413.423013 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.441550 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.442263 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.447093 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.447949 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.449831 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.450692 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.451745 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.452589 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1720821413.453438 2748566 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
[...]
```

I receive numerous messages of 'Skipping the delay kernel, measurement accuracy will be reduced' at the beginning of training. The messages don't affect the training process, however, are new and appear only in the recent version (2.17.0 and tf-nightly).

The flag `TF_CPP_MIN_LOG_LEVEL` doesn't make any difference, so any assistance in understanding or suppressing these messages would be appreciated.

### Standalone code to reproduce the issue

When importing tensorflow it is already possible to see some messages of this type.

```
python -c "import os; os.environ['TF_CPP_MIN_LOG_LEVEL']='3'; import tensorflow as tf;"

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1720831683.868288 3059140 cuda_dnn.cc:8458] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1720831683.872386 3059140 cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
```