Repository: tensorflow
Issue: //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu is flaky
Labels: stat:awaiting tensorflower,type:bug,type:build/install,subtype: ubuntu/linux

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

git HEAD

### Custom code

No

### OS platform and distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current behavior?

//tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu timeouts sometimes

x86 log
https://source.cloud.google.com/results/invocations/75eaa47f-92bd-47d5-9c7a-020a0c67dda9/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5399333727/jobs/9806323589#step:5:7344

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
TIMEOUT: //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu (Summary)
      /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/testlogs/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu/test.log
INFO: From Testing //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu:
==================== Test output for //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu:
2023-07-04 08:53:48.311167: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-04 08:53:48.373167: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] ClusterTypeNameTest.testArbitraryCurrentTaskType
INFO:tensorflow:Using local port 38369
I0704 08:53:53.254792 140378756265792 test_util.py:3796] Using local port 38369
INFO:tensorflow:Using local port 44357
I0704 08:53:53.255300 140378756265792 test_util.py:3796] Using local port 44357
INFO:tensorflow:Using local port 46287
I0704 08:53:53.255505 140378756265792 test_util.py:3796] Using local port 46287
INFO:tensorflow:time(__main__.ClusterTypeNameTest.testArbitraryCurrentTaskType): 0.0s
I0704 08:53:53.256115 140378756265792 test_util.py:2464] time(__main__.ClusterTypeNameTest.testArbitraryCurrentTaskType): 0.0s
[       OK ] ClusterTypeNameTest.testArbitraryCurrentTaskType
[ RUN      ] ClusterTypeNameTest.testArbitraryJobName
INFO:tensorflow:Using local port 43625
I0704 08:53:53.256868 140378756265792 test_util.py:3796] Using local port 43625
INFO:tensorflow:Using local port 38445
I0704 08:53:53.257118 140378756265792 test_util.py:3796] Using local port 38445
INFO:tensorflow:Using local port 33687
I0704 08:53:53.257305 140378756265792 test_util.py:3796] Using local port 33687
INFO:tensorflow:Using local port 43947
I0704 08:53:53.257477 140378756265792 test_util.py:3796] Using local port 43947
INFO:tensorflow:time(__main__.ClusterTypeNameTest.testArbitraryJobName): 0.0s
I0704 08:53:53.258199 140378756265792 test_util.py:2464] time(__main__.ClusterTypeNameTest.testArbitraryJobName): 0.0s
[       OK ] ClusterTypeNameTest.testArbitraryJobName
[ RUN      ] ClusterTypeNameTest.testLessThanOnePs
INFO:tensorflow:Using local port 37051
I0704 08:53:53.258810 140378756265792 test_util.py:3796] Using local port 37051
INFO:tensorflow:Using local port 44491
I0704 08:53:53.259010 140378756265792 test_util.py:3796] Using local port 44491
INFO:tensorflow:time(__main__.ClusterTypeNameTest.testLessThanOnePs): 0.0s
I0704 08:53:53.259491 140378756265792 test_util.py:2464] time(__main__.ClusterTypeNameTest.testLessThanOnePs): 0.0s
[       OK ] ClusterTypeNameTest.testLessThanOnePs
[ RUN      ] ClusterTypeNameTest.testLessThanOneWorker
INFO:tensorflow:Using local port 33303
I0704 08:53:53.260033 140378756265792 test_util.py:3796] Using local port 33303
INFO:tensorflow:Using local port 33109
I0704 08:53:53.260214 140378756265792 test_util.py:3796] Using local port 33109
INFO:tensorflow:time(__main__.ClusterTypeNameTest.testLessThanOneWorker): 0.0s
I0704 08:53:53.260598 140378756265792 test_util.py:2464] time(__main__.ClusterTypeNameTest.testLessThanOneWorker): 0.0s
[       OK ] ClusterTypeNameTest.testLessThanOneWorker
[ RUN      ] ClusterTypeNameTest.testMoreThanOneChief
INFO:tensorflow:Using local port 36459
I0704 08:53:53.261005 140378756265792 test_util.py:3796] Using local port 36459
INFO:tensorflow:Using local port 33799
I0704 08:53:53.261153 140378756265792 test_util.py:3796] Using local port 33799
INFO:tensorflow:Using local port 39497
I0704 08:53:53.261297 140378756265792 test_util.py:3796] Using local port 39497
INFO:tensorflow:Using local port 36657
I0704 08:53:53.261423 140378756265792 test_util.py:3796] Using local port 36657
INFO:tensorflow:Using local port 37059
I0704 08:53:53.261545 140378756265792 test_util.py:3796] Using local port 37059
INFO:tensorflow:time(__main__.ClusterTypeNameTest.testMoreThanOneChief): 0.0s
I0704 08:53:53.261911 140378756265792 test_util.py:2464] time(__main__.ClusterTypeNameTest.testMoreThanOneChief): 0.0s
[       OK ] ClusterTypeNameTest.testMoreThanOneChief
[ RUN      ] ClusterTypeNameTest.test_session
[  SKIPPED ] ClusterTypeNameTest.test_session
INFO:tensorflow:Now creating a MultiProcessCluster with num_workers=2, num_ps=3.
I0704 08:53:53.262382 140378756265792 multi_worker_test_base.py:335] Now creating a MultiProcessCluster with num_workers=2, num_ps=3.
INFO:tensorflow:Using local port 34725
I0704 08:53:53.262561 140378756265792 test_util.py:3796] Using local port 34725
INFO:tensorflow:Using local port 41389
I0704 08:53:53.262695 140378756265792 test_util.py:3796] Using local port 41389
INFO:tensorflow:Using local port 42953
I0704 08:53:53.262822 140378756265792 test_util.py:3796] Using local port 42953
INFO:tensorflow:Using local port 39597
I0704 08:53:53.262954 140378756265792 test_util.py:3796] Using local port 39597
INFO:tensorflow:Using local port 40735
I0704 08:53:53.263074 140378756265792 test_util.py:3796] Using local port 40735
2023-07-04 08:53:54.005149: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-04 08:53:54.092559: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-04 08:53:54.232593: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-04 08:53:54.295324: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
-- Test timed out at 2023-07-04 08:58:46 UTC --
Thread 0x00007fac735de700 (most recent call first):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 258 in _continuously_readline_from_sub
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007fac765a0700 (most recent call first):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 258 in _continuously_readline_from_sub
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007fac025a4700 (most recent call first):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 258 in _continuously_readline_from_sub
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007fabffda3700 (most recent call first):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 527 in _process_watchdog
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007fabff5a2700 (most recent call first):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 258 in _continuously_readline_from_sub
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Current thread 0x00007fac79e6b740 (most recent call first):
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 379 in _recv
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 414 in _recv_bytes
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 250 in recv
  File "/usr/lib/python3.9/multiprocessing/managers.py", line 810 in _callmethod
  File "/usr/lib/python3.9/multiprocessing/managers.py", line 1085 in wait
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_worker_test_base.py", line 270 in start
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_worker_test_base.py", line 348 in create_multi_process_cluster
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/parameter_server_strategy_v2_test.py", line 62 in setUpClass
  File "/usr/lib/python3.9/unittest/suite.py", line 166 in _handleClassSetUp
  File "/usr/lib/python3.9/unittest/suite.py", line 114 in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.9/unittest/runner.py", line 184 in run
  File "/usr/lib/python3.9/unittest/main.py", line 271 in runTests
  File "/usr/lib/python3.9/unittest/main.py", line 101 in __init__
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/absl_py/absl/app.py", line 258 in _run_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/absl_py/absl/app.py", line 312 in run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/eager/test.py", line 25 in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 167 in test_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1455 in test_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/parameter_server_strategy_v2_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/parameter_server_strategy_v2_test.py", line 713 in <module>
```