Repository: tensorflow
Issue: //tensorflow/python/distribute/failure_handling:failure_handler_test is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/distribute/failure_handling:failure_handler_test will timeout sometimes

x86 log
https://source.cloud.google.com/results/invocations/302ca1a4-593b-430c-b278-038351228670/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5363735926/jobs/9731502578#step:5:11034

Looks like a network port conflict.

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
INFO: From Testing //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 4 of 8):
==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 4 of 8):
2023-06-22 16:46:57.933478: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-22 16:46:58.014195: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker
INFO:tensorflow:Using local port 38723
I0622 16:47:02.418146 140436227843904 test_util.py:3796] Using local port 38723
INFO:tensorflow:Using local port 41053
I0622 16:47:02.418907 140436227843904 test_util.py:3796] Using local port 41053
INFO:tensorflow:Using local port 45087
I0622 16:47:02.419116 140436227843904 test_util.py:3796] Using local port 45087
INFO:tensorflow:Using local port 38125
I0622 16:47:02.419290 140436227843904 test_util.py:3796] Using local port 38125
2023-06-22 16:47:03.130184: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-22 16:47:03.185580: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-22 16:47:03.185763: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-22 16:47:03.241253: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO:tensorflow:Cluster starting.
I0622 16:47:04.817156 140436227843904 failure_handler_test.py:432] Cluster starting.
[worker-0]:    I0622 16:47:04.842670 140652189284160 multi_process_runner.py:840] Subprocess with PID 588346 (worker, 0) is now being started.
[worker-0]:    I0622 16:47:04.842952 140652189284160 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38723", "localhost:41053", "localhost:45087", "localhost:38125"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-1]:    I0622 16:47:04.858144 140652189284160 multi_process_runner.py:840] Subprocess with PID 588646 (worker, 1) is now being started.
[worker-1]:    I0622 16:47:04.858554 140652189284160 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38723", "localhost:41053", "localhost:45087", "localhost:38125"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-0]:    2023-06-22 16:47:04.907068: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38723
[worker-2]:    I0622 16:47:04.927917 140652189284160 multi_process_runner.py:840] Subprocess with PID 588933 (worker, 2) is now being started.
[worker-0]:    2023-06-22 16:47:04.943758: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 14520256932767538069
[worker-0]:    2023-06-22 16:47:04.944905: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    I0622 16:47:04.928328 140652189284160 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38723", "localhost:41053", "localhost:45087", "localhost:38125"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-1]:    2023-06-22 16:47:04.995414: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:41053
[worker-0]:    2023-06-22 16:47:04.997341: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 7622272584115739518
[worker-1]:    2023-06-22 16:47:04.998064: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    I0622 16:47:05.006252 140652189284160 multi_process_runner.py:840] Subprocess with PID 589461 (worker, 3) is now being started.
[worker-2]:    E0622 16:47:05.032748121  588933 server_chttp2.cc:40]        {"created":"@1687452425.032699864","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1687452425.032698162","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":342,"referenced_errors":[{"created":"@1687452425.032676237","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:45087"},{"created":"@1687452425.032697070","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1687452425.032694857","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]}
[worker-2]:    2023-06-22 16:47:05.032844: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server
[worker-2]:    2023-06-22 16:47:05.033076: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:703] Could not start gRPC server
[worker-3]:    I0622 16:47:05.006778 140652189284160 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38723", "localhost:41053", "localhost:45087", "localhost:38125"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-2]:    Process _Process-4:
[worker-2]:    Traceback (most recent call last):
[worker-2]:      File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
[worker-2]:        self.run()
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv
[worker-2]:        return self._actual_run()
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl
[worker-2]:        app.run(lambda _: self._run_impl())
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run
[worker-2]:        _run_main(main, args)
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main
[worker-2]:        sys.exit(main(argv))
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in <lambda>
[worker-2]:        app.run(lambda _: self._run_impl())
[worker-2]:      File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
[worker-2]:        self._target(*self._args, **self._kwargs)
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__
[worker-2]:        six.reraise(*info.exc_info)
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise
[worker-2]:        raise value
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained
[worker-2]:        return_value = fn(*args, **kwargs)
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 146, in worker_fn
[worker-2]:        strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy()
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__
[worker-2]:        CollectiveAllReduceExtended(
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__
[worker-2]:        self._initialize_strategy(self._cluster_resolver, devices=devices)
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy
[worker-2]:        self._initialize_multi_worker(cluster_resolver)
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker
[worker-2]:        context.context().ensure_initialized()
[worker-2]:      File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized
[worker-2]:        pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str)
[worker-2]:    tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server
```
</details>