Repository: tensorflow
Issue: //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux,TF 2.13

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/data/experimental/kernel_tests/service:local_workers_test sometimes fails or timeouts.

x86 log
https://source.cloud.google.com/results/invocations/e41a9dd4-19a3-4298-b34f-6a32eca50e08/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5383187293/jobs/9769619751#step:5:8335

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
INFO: From Testing //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test (shard 1 of 24):
==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test (shard 1 of 24):
2023-06-27 23:02:53.397107: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-27 23:02:53.528865: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] LocalTaskGarbageCollectTest.testMultipleEpochsSharedJob_test_mode_eager_tfapiversion_1_numremoteworkers_0
[  SKIPPED ] LocalTaskGarbageCollectTest.testMultipleEpochsSharedJob_test_mode_eager_tfapiversion_1_numremoteworkers_0
[ RUN      ] LocalTaskGarbageCollectTest.testReadFromDeletedTask_test_mode_eager_tfapiversion_1_numremoteworkers_0
[  SKIPPED ] LocalTaskGarbageCollectTest.testReadFromDeletedTask_test_mode_eager_tfapiversion_1_numremoteworkers_0
[ RUN      ] LocalWorkersTest.testAnonymousJobWithDifferentTargetWorkers_test_mode_graph_tfapiversion_2
INFO:tensorflow:Using local port 43055
I0627 23:02:57.097305 139801719478080 test_util.py:3796] Using local port 43055
2023-06-27 23:02:57.099356: I tensorflow/core/data/service/dispatcher_impl.cc:223] Attempting to restore dispatcher state from journal in /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/dbc16dd8be682dbdabf0b589b8e98f16j99fj6xd/tmptm5ix2lv/tf_data_dispatcher_journal
2023-06-27 23:02:57.099421: I tensorflow/core/data/service/dispatcher_impl.cc:230] No journal found. Starting dispatcher from new state.
2023-06-27 23:02:57.099574: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:43055
INFO:tensorflow:Using local port 44899
I0627 23:02:57.099842 139801719478080 test_util.py:3796] Using local port 44899
2023-06-27 23:02:57.101683: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:43055
2023-06-27 23:02:57.101828: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:44899
INFO:tensorflow:Using local port 41615
I0627 23:02:57.102149 139801719478080 test_util.py:3796] Using local port 41615
2023-06-27 23:02:57.103351: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:43055
2023-06-27 23:02:57.103477: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:41615
INFO:tensorflow:Using local port 37691
I0627 23:02:57.103664 139801719478080 test_util.py:3796] Using local port 37691
2023-06-27 23:02:57.104726: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:43055
2023-06-27 23:02:57.104854: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:37691
INFO:tensorflow:Using local port 39883
I0627 23:02:57.107191 139801719478080 test_util.py:3796] Using local port 39883
2023-06-27 23:02:57.584682: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-27 23:02:57.619502: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-27 23:02:57.634234: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-27 23:02:57.788004: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-27 23:02:59.175789: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:43055
2023-06-27 23:02:59.176006: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:39883
INFO:tensorflow:Using local port 34849
I0627 23:02:59.176878 139801719478080 test_util.py:3796] Using local port 34849
2023-06-27 23:02:59.252223: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:43055
2023-06-27 23:02:59.252462: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:34849
INFO:tensorflow:Using local port 46439
I0627 23:02:59.253159 139801719478080 test_util.py:3796] Using local port 46439
2023-06-27 23:02:59.287010: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:43055
2023-06-27 23:02:59.287229: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:46439
/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/ops/dataset_ops.py:458: UserWarning: To make it possible to preserve tf.data options across serialization boundaries, their implementation has moved to be part of the TensorFlow graph. As a consequence, the options value is in general no longer known at graph construction time. Invoking this method in graph mode retains the legacy behavior of the original implementation, but note that the returned value might not reflect the actual value of the options.
  warnings.warn("To make it possible to preserve tf.data options across "
WARNING:tensorflow:From /usr/lib/python3.9/contextlib.py:87: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
W0627 23:02:59.486639 139801719478080 deprecation.py:364] From /usr/lib/python3.9/contextlib.py:87: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
2023-06-27 23:02:59.508867: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:382] MLIR V1 optimization pass is not enabled
-- Test timed out at 2023-06-27 23:07:52 UTC --
Current thread 0x00007f261fd41740 (most recent call first):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1477 in _call_tf_sessionrun
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1384 in _run_fn
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1401 in _do_call
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1394 in _do_run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1214 in _run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 971 in run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2061 in run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2691 in evaluate
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/kernel_tests/test_base.py", line 237 in assertDatasetProduces
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.py", line 238 in testAnonymousJobWithDifferentTargetWorkers
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test
  File "/usr/lib/python3.9/unittest/case.py", line 550 in _callTestMethod
  File "/usr/lib/python3.9/unittest/case.py", line 592 in run
  File "/usr/lib/python3.9/unittest/case.py", line 651 in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.9/unittest/runner.py", line 184 in run
  File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests
  File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/app.py", line 258 in _run_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/app.py", line 312 in run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/eager/test.py", line 25 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 167 in test_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 165 in test_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.py", line 441 in <module>
```
</details>