Repository: tensorflow
Issue: fault_tolerance_test fails on some systems
Labels: type:build/install,subtype: ubuntu/linux,subtype:bazel,TF 2.7

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Source

source

### Tensorflow Version

2.7.1

### Custom Code

No

### OS Platform and Distribution

Linux RHEL 7

### Mobile device

_No response_

### Python version

3.9.6

### Bazel version

3.7.2

### GCC/Compiler version

11.2

### CUDA/cuDNN version

11.4.1 / cuDNN 8.2.2.26

### GPU model and memory

8 * Tesla A100

### Current Behaviour?

```shell
When running the test suite during/after the build the test `//tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test` fails on this system, while it seemingly passes on another system.
It seems to be very flaky or dependent on the number of CPUs/GPUs. This system has 96 cores and 8 GPUs.
Output is something like
> //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test (2/20 cached) FAILED in 18 out of 20 in 13.5s
```


### Standalone code to reproduce the issue

```shell
Run the following build command: `bazel test --config=noaws --config=nogcp --config=nohdfs --compilation_mode=opt --config=opt --subcommands --verbose_failures --jobs=1 --distinct_host_configuration=false --test_output=errors --build_tests_only --local_test_jobs=1 -- //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test`
```


### Relevant log output

```shell
======================================================================
FAIL: testAddWorkerMidJob_test_mode_eager_tfapiversion_2 (__main__.FaultToleranceTest)
FaultToleranceTest.testAddWorkerMidJob_test_mode_eager_tfapiversion_2
testAddWorkerMidJob_test_mode_eager_tfapiversion_2(mode='eager', tf_api_version=2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/software/TensorFlow/2.7.1-foss-2021b-CUDA-11.4.1/lib/python3.9/site-packages/absl/testing/parameterized.py", line 314, in bound_param_test
    return test_method(self, **testcase_params)
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.7.1/foss-2021b-CUDA-11.4.1/tmpBD23m_-bazel-tf/69a307561ec5a7cdace7b5c5a8971ea2/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/fault_tolerance_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 366, in decorated
    execute_test_method()
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.7.1/foss-2021b-CUDA-11.4.1/tmpBD23m_-bazel-tf/69a307561ec5a7cdace7b5c5a8971ea2/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/fault_tolerance_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 349, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/dev/shm/s3248973-EasyBuild/TensorFlow/2.7.1/foss-2021b-CUDA-11.4.1/tmpBD23m_-bazel-tf/69a307561ec5a7cdace7b5c5a8971ea2/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/fault_tolerance_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/fault_tolerance_test.py", line 220, in testAddWorkerMidJob
    self.assertCountEqual(2 * list(range(num_elements)), results)
AssertionError: Element counts were not equal:
```
</details>