Repository: tensorflow
Issue: Failed to synchronize the stop event: CUDA_ERROR_UNKNOWN: unknown error
Labels: type:build/install,subtype: ubuntu/linux,2.17

Hello together,

I am trying to run the new-benchmark AI for my new platform:
https://pypi.org/project/new-ai-benchmark/

My System configuration is as follows:
*  TF Version: 2.17.0
*  Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.35
*  CPU: 13th Gen Intel(R) Core(TM) i9-13900E
*  CPU RAM: 62 GB
*  GPU/0: NVIDIA L4
*  GPU RAM: 20.3 GB
*  CUDA Version: 12.4
*  CUDA Build: V12.4.131

arun@arun:~$ nvidia-smi
Wed Sep 25 12:52:43 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:01:00.0 Off |                    0 |
| N/A   55C    P8             13W /   72W |      14MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1373      G   /usr/lib/xorg/Xorg                              4MiB |
+-----------------------------------------------------------------------------------------+


While executing the program, at any point of time while running, it fails to complete.
```

2024-09-25 10:02:40.968881: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:1857] failed to synchronize the stop event: CUDA_ERROR_UNKNOWN: unknown error
E0000 00:00:1727251360.968902    4778 gpu_timer.cc:162] INTERNAL: Error destroying CUDA event: CUDA_ERROR_UNKNOWN: unknown error
E0000 00:00:1727251360.968910    4778 gpu_timer.cc:168] INTERNAL: Error destroying CUDA event: CUDA_ERROR_UNKNOWN: unknown error
2024-09-25 10:02:40.969770: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6053): 'status'
[[{{node resnet_v2_152/block2/unit_8/bottleneck_v2/conv2/Conv2D}}]]
2024-09-25 10:02:40.969784: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6053): 'status'
[[{{node resnet_v2_152/block2/unit_8/bottleneck_v2/conv2/Conv2D}}]]
[[output/_3]]
2024-09-25 10:02:40.969798: I tensorflow/core/framework/local_rendezvous.cc:423] Local rendezvous recv item cancelled. Key hash: 16128423615390660009
2024-09-25 10:02:40.992816: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleUnload(it.second)' failed with 'CUDA_ERROR_UNKNOWN'

2024-09-25 10:02:40.992867: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleUnload(it.second)' failed with 'CUDA_ERROR_UNKNOWN'

2024-09-25 10:02:40.992874: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleUnload(it.second)' failed with 'CUDA_ERROR_UNKNOWN'

2024-09-25 10:02:40.992880: W tensorflow/compiler/mlir/tools/kernel_gen/tf_gpu_runtime_wrappers.cc:40] 'cuModuleUnload(it.second)' failed with 'CUDA_ERROR_UNKNOWN'

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tensorflow/python/client/session.py", line 1401, in _do_call
    return fn(*args)
  File "/usr/lib/python3/dist-packages/tensorflow/python/client/session.py", line 1384, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/usr/lib/python3/dist-packages/tensorflow/python/client/session.py", line 1477, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
```

In another instance, there is another reason for failure:

```
Failed to enqueue async memcpy from device to host: CUDA_ERROR_UNKNOWN: unknown error; host dst: 0x763a34bfcf48; GPU src: 0x76387c443400; size: 8=0x8
2024-09-25 11:53:30.387757: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:1595] failed to free device memory at 0x76387c443400; result: CUDA_ERROR_UNKNOWN: unknown error
2024-09-25 11:53:30.387762: W tensorflow/core/kernels/gpu_utils.cc:88] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to memcpy from device to host.'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2024-09-25 11:53:30.388810: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6053): 'status'
[[{{node resnet_v2_50/block3/unit_1/bottleneck_v2/shortcut/Conv2D}}]]
2024-09-25 11:53:30.388848: F ./tensorflow/core/kernels/conv_2d_gpu.h:1028] Non-OK-status: GpuLaunchKernel( SwapDimension1And2InTensor3UsingTiles<T, kNumThreads, kTileSize, kTileSize, conjugate>, total_tiles_count, kNumThreads, 0, d.stream(), input, input_dims, output)
Status: INTERNAL: unknown error
```

Could you please help in resolving the issue? 

Thank you
Arun