Repository: tensorflow
Issue: //tensorflow/python/distribute:vars_test_2gpu is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/distribute:vars_test_2gpu timeouts sometimes.

x86 log
https://source.cloud.google.com/results/invocations/769764d8-8dc9-46fa-a284-78062efe3bd9/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5363572382/jobs/9731245377#step:5:9313

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=rbe --config=pycpp --config=build_event_export
```


### Relevant log output

```shell
[ RUN      ] SyncOnReadScatterReplicaTest.testScatterMax_test_aggregation_VariableAggregationMEAN_distribution_MultiWorkerMirrored2x1CPU_mode_graph_usevarpolicy_True
W0628 01:02:37.513197 140343714740032 context.py:773] Configuring coordination service type may not be effective because the context is already initialized.
WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled.
W0628 01:02:37.513532 140343714740032 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled.
INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',)
I0628 01:02:37.516878 140343714740032 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',)
INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
I0628 01:02:37.517337 140343714740032 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
[chief-0]:     W0628 01:02:37.520201 140325400614720 context.py:773] Configuring coordination service type may not be effective because the context is already initialized.
I0628 01:02:37.521484 140343714740032 multi_process_runner.py:989] Waiting for the result from chief-0
[worker-0]:    W0628 01:02:37.523296 140325400614720 context.py:773] Configuring coordination service type may not be effective because the context is already initialized.
[chief-0]:     W0628 01:02:37.521138 140325400614720 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors.
[worker-0]:    W0628 01:02:37.524588 140325400614720 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors.
[chief-0]:     INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:chief/replica:0/task:0/device:CPU:0', '/job:chief/replica:0/task:0/device:CPU:1']
[chief-0]:     I0628 01:02:37.525059 140325400614720 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:chief/replica:0/task:0/device:CPU:0', '/job:chief/replica:0/task:0/device:CPU:1']
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1']
[worker-0]:    I0628 01:02:37.527691 140325400614720 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1']
[chief-0]:     INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:CPU:0',)
[chief-0]:     I0628 01:02:37.529053 140325400614720 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:chief/task:0/device:CPU:0',)
[chief-0]:     INFO:tensorflow:Check health not enabled.
[chief-0]:     I0628 01:02:37.529462 140325400614720 collective_all_reduce_strategy.py:574] Check health not enabled.
[chief-0]:     INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:17555'], 'worker': ['localhost:24042']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[chief-0]:     I0628 01:02:37.529720 140325400614720 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:17555'], 'worker': ['localhost:24042']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    I0628 01:02:37.531570 140325400614720 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    I0628 01:02:37.532161 140325400614720 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:17555'], 'worker': ['localhost:24042']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0628 01:02:37.532691 140325400614720 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:17555'], 'worker': ['localhost:24042']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    2023-06-28 01:02:37.540481: I tensorflow/core/distributed_runtime/master.cc:240] Scanning workers for devices: 1 total workers
[chief-0]:     2023-06-28 01:02:37.541763: I tensorflow/core/distributed_runtime/master.cc:240] Scanning workers for devices: 1 total workers
-- Test timed out at 2023-06-28 01:07:29 UTC --
Thread 0x00007fa429ef8700 (most recent call first):
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 258 in _continuously_readline_from_sub
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007fa42a779700 (most recent call first):
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 527 in _process_watchdog
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007fa44d0fc700 (most recent call first):
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 258 in _continuously_readline_from_sub
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Current thread 0x00007fa451437740 (most recent call first):
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 379 in _recv
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 414 in _recv_bytes
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 250 in recv
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 991 in run
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580 in decorator
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test
  File "/usr/lib/python3.9/unittest/case.py", line 550 in _callTestMethod
  File "/usr/lib/python3.9/unittest/case.py", line 592 in run
  File "/usr/lib/python3.9/unittest/case.py", line 651 in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.9/unittest/runner.py", line 184 in run
  File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests
  File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/app.py", line 258 in _run_main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/absl_py/absl/app.py", line 312 in run
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/test.py", line 25 in main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 167 in test_main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1455 in test_main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/test_util.py", line 138 in main
  File "/b/f/w/bazel-out/k8-opt/bin/tensorflow/python/distribute/vars_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/vars_test.py", line 1336 in <module>
```
</details>