Repository: tensorflow
Issue: //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

git HEAD

### Custom code

No

### OS platform and distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current behavior?

//tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test fails occasionally

x86 log
https://source.cloud.google.com/results/invocations/0bc426bf-e5ae-4fb6-993f-6199c00d1139/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5411569978/jobs/9834485829#step:5:8675

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
======================================================================
ERROR: testMultipleTags_test_mode_eager_tfapiversion_2 (__main__.WorkerTagsTest)
WorkerTagsTest.testMultipleTags_test_mode_eager_tfapiversion_2
testMultipleTags_test_mode_eager_tfapiversion_2(mode='eager', tf_api_version=2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
    return test_method(self, **testcase_params)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
    execute_test_method()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.py", line 179, in testMultipleTags
    cluster = multi_process_cluster.MultiProcessCluster(
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 81, in __init__
    self._start_local_workers(num_local_workers, worker_tags)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 100, in _start_local_workers
    self.start_local_worker(worker_tags)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 114, in start_local_worker
    worker.start()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/test_base.py", line 110, in start
    self._server.start()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 415, in start
    self._server.start()
NotImplementedError: Failed to get dispatcher version from dispatcher running at localhost:45069:

----------------------------------------------------------------------
Ran 6 tests in 6.597s

FAILED (errors=1)
Exception ignored in: <function MultiProcessCluster.__del__ at 0x7f20f09658b0>
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 160, in __del__
    self._stop()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 155, in _stop
    for (_, worker_process) in self._remote_workers:
AttributeError: 'MultiProcessCluster' object has no attribute '_remote_workers'
2023-07-02 09:48:51.833944: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 45069
2023-07-02 09:48:51.934278: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 44313
================================================================================
```