Repository: tensorflow
Issue: //tensorflow/python/distribute/experimental/rpc:rpc_ops_test is flaky
Labels: stat:awaiting tensorflower,type:bug,subtype: ubuntu/linux,TF 2.13

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

git HEAD

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/Compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current Behaviour?

//tensorflow/python/distribute/experimental/rpc:rpc_ops_test sometimes fails.

x86 log
https://source.cloud.google.com/results/invocations/e7ac5e31-66d5-4f2e-a095-045cb52cc20f/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5348094575/jobs/9697499180#step:5:8263

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
FAIL: //tensorflow/python/distribute/experimental/rpc:rpc_ops_test (shard 4 of 7) (see /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/testlogs/tensorflow/python/distribute/experimental/rpc/rpc_ops_test/shard_4_of_7/test.log)
INFO: From Testing //tensorflow/python/distribute/experimental/rpc:rpc_ops_test (shard 4 of 7):
==================== Test output for //tensorflow/python/distribute/experimental/rpc:rpc_ops_test (shard 4 of 7):
2023-06-27 21:51:01.865535: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-27 21:51:01.915964: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] RpcOpsTest.test_client_timeout
E0627 21:51:07.067311728 3191958 server_chttp2.cc:40]        {"created":"@1687902667.067282680","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":404,"referenced_errors":[{"created":"@1687902667.067278835","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:34603"}]}
2023-06-27 21:51:07.067457: I tensorflow/distribute/experimental/rpc/kernels/rpc_ops.cc:347] Server listening on: localhost:34603
2023-06-27 21:51:08.565889: I tensorflow/distribute/experimental/rpc/kernels/rpc_ops.cc:314] Shutting down server listening on: localhost:34603
INFO:tensorflow:time(__main__.RpcOpsTest.test_client_timeout): 2.71s
I0627 21:51:08.574481 139721505019712 test_util.py:2464] time(__main__.RpcOpsTest.test_client_timeout): 2.71s
[  FAILED  ] RpcOpsTest.test_client_timeout
[ RUN      ] RpcOpsTest.test_queue_resource
/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.runfiles/org_tensorflow/tensorflow/python/saved_model/nested_structure_coder.py:458: UserWarning: Encoding a StructuredValue with type tf.NoneTensorSpec; loading this StructuredValue will require that this type be imported and registered.
  warnings.warn("Encoding a StructuredValue with type %s; loading this "
E0627 21:51:08.630379336 3163442 server_chttp2.cc:40]        {"created":"@1687902668.630352103","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":404,"referenced_errors":[{"created":"@1687902668.630348956","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:43377"}]}
2023-06-27 21:51:08.630532: I tensorflow/distribute/experimental/rpc/kernels/rpc_ops.cc:347] Server listening on: localhost:43377
2023-06-27 21:51:08.686495: I tensorflow/distribute/experimental/rpc/kernels/rpc_ops.cc:314] Shutting down server listening on: localhost:43377
INFO:tensorflow:time(__main__.RpcOpsTest.test_queue_resource): 0.14s
I0627 21:51:08.712013 139721505019712 test_util.py:2464] time(__main__.RpcOpsTest.test_queue_resource): 0.14s
[       OK ] RpcOpsTest.test_queue_resource
[ RUN      ] RpcOpsTest.test_rpc_ops_non_blocking_convenience_methods
E0627 21:51:08.774584099 3163442 server_chttp2.cc:40]        {"created":"@1687902668.774551355","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":404,"referenced_errors":[{"created":"@1687902668.774547071","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:39177"}]}
2023-06-27 21:51:08.774736: I tensorflow/distribute/experimental/rpc/kernels/rpc_ops.cc:347] Server listening on: localhost:39177
2023-06-27 21:51:08.798979: I tensorflow/distribute/experimental/rpc/kernels/rpc_ops.cc:314] Shutting down server listening on: localhost:39177
INFO:tensorflow:time(__main__.RpcOpsTest.test_rpc_ops_non_blocking_convenience_methods): 0.09s
I0627 21:51:08.802805 139721505019712 test_util.py:2464] time(__main__.RpcOpsTest.test_rpc_ops_non_blocking_convenience_methods): 0.09s
[       OK ] RpcOpsTest.test_rpc_ops_non_blocking_convenience_methods
======================================================================
ERROR: test_client_timeout (__main__.RpcOpsTest)
RpcOpsTest.test_client_timeout
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.runfiles/org_tensorflow/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.py", line 487, in test_client_timeout
    client = rpc_ops.GrpcClient(
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.runfiles/org_tensorflow/tensorflow/python/distribute/experimental/rpc/rpc_ops.py", line 342, in __init__
    self._client_handle, methods = gen_rpc_ops.rpc_client(
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.runfiles/org_tensorflow/tensorflow/distribute/experimental/rpc/kernels/gen_rpc_ops.py", line 333, in rpc_client
    return rpc_client_eager_fallback(
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.runfiles/org_tensorflow/tensorflow/distribute/experimental/rpc/kernels/gen_rpc_ops.py", line 407, in rpc_client_eager_fallback
    _result = _execute.execute(b"RpcClient", 2, inputs=_inputs_flat,
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/distribute/experimental/rpc/rpc_ops_test.runfiles/org_tensorflow/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnavailableError: {{function_node __wrapped__RpcClient_device_/job:localhost/replica:0/task:0/device:CPU:0}} GOAWAY received
Additional GRPC error information while calling /tensorflow.rpc.RpcService/List:
:{"created":"@1687902668.574079798","description":"Error received from peer ipv4:127.0.0.1:34603","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"GOAWAY received","grpc_status":14} [Op:RpcClient]
```
</details>