Repository: tensorflow
Issue: //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test is flaky
Labels: stat:awaiting tensorflower,type:build/install,subtype: ubuntu/linux

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

git HEAD

### Custom code

No

### OS platform and distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.16

### Bazel version

6.1.0

### GCC/compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current behavior?

//tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test fails occasionally

x86 log
https://source.cloud.google.com/results/invocations/c5169019-316f-43e1-8310-67596d2aae9a/log

AARCH64 log
https://github.com/tensorflow/tensorflow/actions/runs/5330158020/jobs/9656553237#step:5:8169

### Standalone code to reproduce the issue

```shell
bazel --bazelrc=/usertools/cpu.bazelrc test --config=pycpp --config=build_event_export --remote_cache=https://storage.googleapis.com/tensorflow-devinfra-bazel-cache/norbe --google_default_credentials
```


### Relevant log output

```shell
FAIL: //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test (shard 7 of 17) (see /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test/shard_7_of_17/test.log)
INFO: From Testing //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test (shard 7 of 17):
==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test (shard 7 of 17):
2023-07-04 13:14:35.474115: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-07-04 13:14:35.591768: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running tests under Python 3.9.17: /usr/bin/python3
[ RUN      ] SnapshotFtTest.testLargeMultiSourceSnapshotRecoversAndCompletes_test_mode_graph_tfapiversion_1
[  SKIPPED ] SnapshotFtTest.testLargeMultiSourceSnapshotRecoversAndCompletes_test_mode_graph_tfapiversion_1
[ RUN      ] SnapshotFtTest.testRepeatedDatasetRecoversAndCompletes_test_mode_graph_tfapiversion_2
2023-07-04 13:14:39.585700: I tensorflow/core/data/service/dispatcher_impl.cc:223] Attempting to restore dispatcher state from journal in /root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/b16384d7dce82bdf5680b78ace436b42jph8vvz5/tmpt0datbl4/tf_data_dispatcher_journal
2023-07-04 13:14:39.585733: I tensorflow/core/data/service/dispatcher_impl.cc:230] No journal found. Starting dispatcher from new state.
2023-07-04 13:14:39.585895: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:46609
2023-07-04 13:14:39.587379: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:46609
2023-07-04 13:14:39.587535: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:43947
2023-07-04 13:14:39.588806: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:46609
2023-07-04 13:14:39.588951: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:43935
2023-07-04 13:14:39.590017: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:46609
2023-07-04 13:14:39.590160: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:46577
WARNING:tensorflow:From /usr/lib/python3.9/contextlib.py:87: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
W0704 13:14:39.597163 140076266616640 deprecation.py:364] From /usr/lib/python3.9/contextlib.py:87: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
2023-07-04 13:14:39.618944: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:382] MLIR V1 optimization pass is not enabled
2023-07-04 13:14:39.670608: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at distributed_save_op.cc:103 : UNIMPLEMENTED: Failed to get dispatcher version from dispatcher running at localhost:46609:
ERROR:tensorflow:Graph execution error:

Detected at node 'DistributedSave' defined at (most recent call last):
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 473, in <module>
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result
    File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__
    File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests
    File "/usr/lib/python3.9/unittest/runner.py", line 184, in run
    File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
    File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
    File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
    File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
    File "/usr/lib/python3.9/unittest/case.py", line 651, in __call__
    File "/usr/lib/python3.9/unittest/case.py", line 592, in run
    File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 357, in testRepeatedDatasetRecoversAndCompletes
Node: 'DistributedSave'
Failed to get dispatcher version from dispatcher running at localhost:46609:
	 [[{{node DistributedSave}}]]

Original stack trace for 'DistributedSave':
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 473, in <module>
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 312, in run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 258, in _run_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result
  File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__
  File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests
  File "/usr/lib/python3.9/unittest/runner.py", line 184, in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
  File "/usr/lib/python3.9/unittest/case.py", line 651, in __call__
  File "/usr/lib/python3.9/unittest/case.py", line 592, in run
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 357, in testRepeatedDatasetRecoversAndCompletes
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/ops/distributed_save_op.py", line 56, in distributed_save
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/ops/gen_experimental_dataset_ops.py", line 2041, in distributed_save
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 2647, in _create_op_internal
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1152, in from_node_def

E0704 13:14:39.671662 140076266616640 test_util.py:2067] Graph execution error:

Detected at node 'DistributedSave' defined at (most recent call last):
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 473, in <module>
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result
    File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__
    File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests
    File "/usr/lib/python3.9/unittest/runner.py", line 184, in run
    File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
    File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
    File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
    File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
    File "/usr/lib/python3.9/unittest/case.py", line 651, in __call__
    File "/usr/lib/python3.9/unittest/case.py", line 592, in run
    File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 357, in testRepeatedDatasetRecoversAndCompletes
Node: 'DistributedSave'
Failed to get dispatcher version from dispatcher running at localhost:46609:
	 [[{{node DistributedSave}}]]

Original stack trace for 'DistributedSave':
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 473, in <module>
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 312, in run
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/app.py", line 258, in _run_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result
  File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__
  File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests
  File "/usr/lib/python3.9/unittest/runner.py", line 184, in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
  File "/usr/lib/python3.9/unittest/suite.py", line 84, in __call__
  File "/usr/lib/python3.9/unittest/suite.py", line 122, in run
  File "/usr/lib/python3.9/unittest/case.py", line 651, in __call__
  File "/usr/lib/python3.9/unittest/case.py", line 592, in run
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.py", line 357, in testRepeatedDatasetRecoversAndCompletes
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/ops/distributed_save_op.py", line 56, in distributed_save
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/ops/gen_experimental_dataset_ops.py", line 2041, in distributed_save
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 2647, in _create_op_internal
  File "/root/.cache/bazel/_bazel_root/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/distributed_save_ft_test.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1152, in from_node_def

[  FAILED  ] SnapshotFtTest.testRepeatedDatasetRecoversAndCompletes_test_mode_graph_tfapiversion_2
```