Repository: tensorflow
Issue: //tensorflow/python/distribute:moving_averages_test is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/distribute:moving_averages_test fails sometimes.

x86 log
https://source.cloud.google.com/results/invocations/1e048065-76db-4dab-b1a5-093dd542b27d/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5383507734/jobs/9770356325#step:5:8417

Looks like a network port conflict issue

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
INFO: From Testing //tensorflow/python/distribute:moving_averages_test_cpu (shard 3 of 5):
==================== Test output for //tensorflow/python/distribute:moving_averages_test_cpu (shard 3 of 5):
2023-06-28 10:24:27.228811: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-28 10:24:27.325938: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] AssignMovingAveragesTest.testAssignVariable_test_distribution_MirroredCPUAndGPU_mode_graph
INFO:tensorflow:time(__main__.AssignMovingAveragesTest.testAssignVariable_test_distribution_MirroredCPUAndGPU_mode_graph): 0.03s
I0628 10:24:32.088060 139996622530368 test_util.py:2464] time(__main__.AssignMovingAveragesTest.testAssignVariable_test_distribution_MirroredCPUAndGPU_mode_graph): 0.03s
[  SKIPPED ] AssignMovingAveragesTest.testAssignVariable_test_distribution_MirroredCPUAndGPU_mode_graph
[ RUN      ] AssignMovingAveragesTest.testAssignVariable_test_distribution_MultiWorkerMirrored4x1CPU_mode_graph
W0628 10:24:32.126455 139996622530368 context.py:773] Configuring coordination service type may not be effective because the context is already initialized.
WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled.
W0628 10:24:32.126971 139996622530368 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled.
INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',)
I0628 10:24:32.164073 139996622530368 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',)
INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
I0628 10:24:32.171577 139996622530368 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
INFO:tensorflow:Using local port 43531
I0628 10:24:32.173116 139996622530368 test_util.py:3796] Using local port 43531
INFO:tensorflow:Using local port 34115
I0628 10:24:32.173517 139996622530368 test_util.py:3796] Using local port 34115
INFO:tensorflow:Using local port 34873
I0628 10:24:32.173699 139996622530368 test_util.py:3796] Using local port 34873
INFO:tensorflow:Using local port 40455
I0628 10:24:32.173828 139996622530368 test_util.py:3796] Using local port 40455
2023-06-28 10:24:32.849027: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-28 10:24:32.904538: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 10:24:32.917529: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-28 10:24:32.995322: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
[chief-0]:     I0628 10:24:34.534415 140662937323328 multi_process_runner.py:840] Subprocess with PID 1244 (chief, 0) is now being started.
[chief-0]:     I0628 10:24:34.534823 140662937323328 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"chief": ["localhost:43531"], "worker": ["localhost:34115", "localhost:34873", "localhost:40455"]}, "task": {"type": "chief", "index": 0}, "rpc_layer": "grpc"}'
I0628 10:24:34.561606 139996622530368 multi_process_runner.py:989] Waiting for the result from chief-0
[worker-1]:    I0628 10:24:34.596723 140662937323328 multi_process_runner.py:840] Subprocess with PID 1479 (worker, 1) is now being started.
[worker-1]:    I0628 10:24:34.597120 140662937323328 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"chief": ["localhost:43531"], "worker": ["localhost:34115", "localhost:34873", "localhost:40455"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-1]:    2023-06-28 10:24:34.655212: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34873
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0']
[worker-1]:    I0628 10:24:34.661444 140662937323328 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0']
[chief-0]:     E0628 10:24:34.621842982    1244 server_chttp2.cc:40]        {"created":"@1687947874.621805165","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1687947874.621803913","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":342,"referenced_errors":[{"created":"@1687947874.621786452","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:43531"},{"created":"@1687947874.621803346","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1687947874.621800946","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]}
[chief-0]:     2023-06-28 10:24:34.621944: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server
[worker-0]:    I0628 10:24:34.608072 140662937323328 multi_process_runner.py:840] Subprocess with PID 1472 (worker, 0) is now being started.
[chief-0]:     2023-06-28 10:24:34.622255: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:703] Could not start gRPC server
[chief-0]:     Process _Process-3:
[chief-0]:     Traceback (most recent call last):
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/strategy_combinations.py", line 207, in skip_if_cannot_start_grpc_server
[chief-0]:         return _create_multi_worker_mirrored()
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/strategy_combinations.py", line 189, in _create_multi_worker_mirrored
[chief-0]:         strategy = CollectiveAllReduceStrategy(cluster_resolver=resolver)
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__
[chief-0]:         CollectiveAllReduceExtended(
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__
[chief-0]:         self._initialize_strategy(self._cluster_resolver, devices=devices)
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy
[chief-0]:         self._initialize_multi_worker(cluster_resolver)
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker
[chief-0]:         context.context().ensure_initialized()
[chief-0]:       File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/moving_averages_test_cpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized
[chief-0]:         pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str)
[chief-0]:     tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server
```
</details>