Repository: tensorflow
Issue: Tensorflow Profiler does not work on WSL2: Failed to load libcupti (is it installed and accessible?) 
Labels: stat:awaiting tensorflower,type:bug,type:others,wsl2

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

v2.12.0-rc1-12-g0db597d0d75 2.12.0

### Custom code

Yes

### OS platform and distribution

Windows 10 WSL Ubuntu

### Mobile device

Ubuntu 22.04

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8/8.6

### GPU model and memory

_No response_

### Current behavior?

After following exactly the steps mentioned in https://www.tensorflow.org/install/pip for installing Tensorflow on WSL2, and installing the latest version of the profiler plugin, the Tensorboard profiler does not seem to work.

This is with a fresh WSL2 install, miniconda install, etc.

![image](https://github.com/tensorflow/tensorflow/assets/11645696/adbaf3c1-4f03-43fe-80d7-39c484ac91a7)

```
Failed to load libcupti (is it installed and accessible?) 

No step marker observed and hence the step time is unknown. This may happen if (1) training steps are not instrumented (e.g., if you are not using Keras) or (2) the profiling duration is shorter than the step time. For (1), you need to add step instrumentation; for (2), you may try to profile longer. 
```

The problem does not seem to be that lubcupti fails to load (despite what is indicated by Tensorboard); libcupti seems to be found just fine, but there may be some problems - See the attached log output for possible clues as to what's happening. 

### Standalone code to reproduce the issue

```shell
# The below code is copied directly from https://github.com/keras-team/keras-io/blob/master/examples/vision/mnist_convnet.py - with the single addition of adding Tensorboard profiling.

import numpy as np
from tensorflow import keras
from tensorflow.keras import layers

"""
## Prepare the data
"""

# Model / data parameters
num_classes = 10
input_shape = (28, 28, 1)

# Load the data and split it between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
print("x_train shape:", x_train.shape)
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")


# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

"""
## Build the model
"""

model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

model.summary()

"""
## Train the model
"""

batch_size = 128
epochs = 15

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1, callbacks=[
    keras.callbacks.TensorBoard(profile_batch=[20, 30])
])

"""
## Evaluate the trained model
"""

score = model.evaluate(x_test, y_test, verbose=0)
print("Test loss:", score[0])
print("Test accuracy:", score[1])
```


### Relevant log output

```shell
17/422 [>.............................] - ETA: 2s - loss: 2.0755 - accuracy: 0.38602023-07-07 15:45:09.034256: I tensorflow/tsl/profiler/lib/profiler_session.cc:104] Profiler session initializing.
2023-07-07 15:45:09.034284: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started.
2023-07-07 15:45:09.034300: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:135] cuptiGetTimestamp: ignored due to a previous error.
2023-07-07 15:45:09.034304: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:186] cuptiSubscribe: ignored due to a previous error.
2023-07-07 15:45:09.034307: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:459] cuptiGetResultString: ignored due to a previous error.
2023-07-07 15:45:09.034309: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1730] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error 
 25/422 [>.............................] - ETA: 2s - loss: 1.8839 - accuracy: 0.46882023-07-07 15:45:09.105818: I tensorflow/tsl/profiler/lib/profiler_session.cc:70] Profiler session collecting data.
2023-07-07 15:45:09.105940: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:142] cuptiFinalize: ignored due to a previous error.
2023-07-07 15:45:09.105943: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:459] cuptiGetResultString: ignored due to a previous error.
2023-07-07 15:45:09.105945: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1822] function cupti_interface_->Finalize()failed with error 
2023-07-07 15:45:09.107652: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:135] cuptiGetTimestamp: ignored due to a previous error.
2023-07-07 15:45:09.107660: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:135] cuptiGetTimestamp: ignored due to a previous error.
2023-07-07 15:45:09.107663: I tensorflow/compiler/xla/backends/profiler/gpu/cupti_collector.cc:541]  GpuTracer has collected 0 callback api events and 0 activity events. 
2023-07-07 15:45:09.107809: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down.
```