Repository: pytorch
Issue: CI fails for test_compare_cpu_nn_functional_embedding_cuda_float32 which is not reproducible locally
Labels: module: cuda,module: tests,triaged,module: embedding

### 🐛 Describe the bug

The PR https://github.com/pytorch/pytorch/pull/85011 to add the test `test_compare_cpu` compares GPU and CPU results. 

In the CI workflow, the test `test_compare_cpu` with the op `nn.functional.embedding` fails with this error message:
```
======================================================================
FAIL [0.009s]: test_compare_cpu_nn_functional_embedding_cuda_float32 (__main__.TestCommonCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1950, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 378, in instantiated_test
    result = test(self, **param_kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 815, in test_wrapper
    return test(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 975, in only_fn
    return fn(slf, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1283, in wrapper
    fn(*args, **kwargs)
  File "/var/lib/jenkins/workspace/test/test_ops.py", line 187, in test_compare_cpu
    self.assertEqual(cuda_results, cpu_results)
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2412, in assertEqual
    assert_equal(
  File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

Mismatched elements: 15 / 45 (33.3%)
Greatest absolute difference: 0.5051435921341181 at index (0, 0, 3) (up to 1e-05 allowed)
Greatest relative difference: 0.9674892692698572 at index (0, 0, 2) (up to 1.3e-06 allowed)
```

However when I run this on `us.quansight.dev`. I can not reproduce the error and I get an unexpected success

```
================================================== 434 passed, 501 skipped, 47841 deselected, 26 xfailed, 11 warnings in 34.78s ==================================================
(pytorch2) srossross@qgpu1:~/pytorch$ py.test test/test_ops.py -v -k test_compare_cpu_nn_functional_embedding_cuda_float32
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /home/srossross/.conda/envs/pytorch2/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/srossross/pytorch/.hypothesis/examples')
rootdir: /home/srossross/pytorch, configfile: pytest.ini
plugins: hypothesis-6.54.4
collected 48802 items / 48801 deselected / 1 selected                                                                                                                            

test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_embedding_cuda_float32 FAILED                                                                             [100%]

==================================================================================== FAILURES ====================================================================================
______________________________________________________ TestCommonCUDA.test_compare_cpu_nn_functional_embedding_cuda_float32 ______________________________________________________
Unexpected success
============================================================================ short test summary info =============================================================================
FAILED test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_embedding_cuda_float32
=============================================================== 1 failed, 48801 deselected, 11 warnings in 15.09s ================================================================
```

### Versions

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.9.0 (default, Nov 15 2020, 14:28:56)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-147-generic-x86_64-with-glibc2.27
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-10.2.89/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.6
[conda] numpy                     1.21.6                   pypi_0    pypi


cc @ngimel @mruberry