Repository: tensorflow
Issue: //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux,subtype:bazel

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test will sometimes timeout.

x86 log
https://source.cloud.google.com/results/invocations/75230523-dc04-40ad-bcf3-06df3b94a119/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5383235016/jobs/9769729147#step:5:8442

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
TIMEOUT: //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test (Summary)
      /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test/shard_6_of_32/test.log
INFO: From Testing //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test (shard 6 of 32):
==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test (shard 6 of 32):
2023-06-28 08:01:34.863480: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-28 08:01:34.953626: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] AutoShardTest.testBatchDataset_test_mode_eager_tfapiversion_1_shardingpolicy_ShardingPolicyFILEORDATA
[  SKIPPED ] AutoShardTest.testBatchDataset_test_mode_eager_tfapiversion_1_shardingpolicy_ShardingPolicyFILEORDATA
[ RUN      ] AutoShardTest.testEnumerateShardingPolicies_test_mode_graph_tfapiversion_1_shardingpolicy_ShardingPolicyDYNAMIC
[  SKIPPED ] AutoShardTest.testEnumerateShardingPolicies_test_mode_graph_tfapiversion_1_shardingpolicy_ShardingPolicyDYNAMIC
[ RUN      ] AutoShardTest.testRangeDataset_AutoShard_test_mode_eager_tfapiversion_1_shardingpolicy_ShardingPolicyFILEORDATA
[  SKIPPED ] AutoShardTest.testRangeDataset_AutoShard_test_mode_eager_tfapiversion_1_shardingpolicy_ShardingPolicyFILEORDATA
[ RUN      ] AutoShardTest.testRangeDataset_ShardHintUsedInWrongShardingPolicy_test_mode_eager_tfapiversion_1_shardingpolicy_ShardingPolicyOFF
[  SKIPPED ] AutoShardTest.testRangeDataset_ShardHintUsedInWrongShardingPolicy_test_mode_eager_tfapiversion_1_shardingpolicy_ShardingPolicyOFF
[ RUN      ] AutoShardTest.testTFRecordDataset_FewerFilesThanWorkers_DataShard_test_mode_eager_tfapiversion_2
INFO:tensorflow:Using local port 35531
I0628 08:02:03.552320 139750982424384 test_util.py:3796] Using local port 35531
2023-06-28 08:02:03.553738: I tensorflow/core/data/service/dispatcher_impl.cc:223] Attempting to restore dispatcher state from journal in /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/df4595e40f65712a12ed239eda341855ao10glmf/tmplcxh6ash/tf_data_dispatcher_journal
2023-06-28 08:02:03.553799: I tensorflow/core/data/service/dispatcher_impl.cc:230] No journal found. Starting dispatcher from new state.
2023-06-28 08:02:03.553987: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:35531
INFO:tensorflow:Using local port 43623
I0628 08:02:03.556421 139750982424384 test_util.py:3796] Using local port 43623
2023-06-28 08:02:04.452927: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-28 08:02:04.495706: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-28 08:02:04.557281: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 08:02:04.586634: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 08:02:06.406403: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:35531
2023-06-28 08:02:06.406887: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:43623
INFO:tensorflow:Using local port 40059
I0628 08:02:06.407576 139750982424384 test_util.py:3796] Using local port 40059
E0628 08:02:06.423661297  858602 server_chttp2.cc:40]        {"created":"@1687939326.423469587","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1687939326.423463864","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":342,"referenced_errors":[{"created":"@1687939326.423438207","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:40059"},{"created":"@1687939326.423463182","description":"Unable to configure socket","fd":8,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1687939326.423455518","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]}
Process _RemoteWorkerProcess-2:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl
    app.run(lambda _: self._run_impl())
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/absl_py/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/absl_py/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in <lambda>
    app.run(lambda _: self._run_impl())
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 42, in run
    self.start_worker()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 50, in start_worker
    self._worker.start()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/test_base.py", line 110, in start
    self._server.start()
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/auto_shard_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/service/server_lib.py", line 415, in start
    self._server.start()
RuntimeError: Could not start gRPC server
-- Test timed out at 2023-06-28 08:06:34 UTC --
```
</details>