Repository: pytorch
Issue: PyTorch 1.12, high failure rate for test_optim/test_nadam
Labels: module: optimizer,triaged

### 🐛 Describe the bug

Similar to https://github.com/pytorch/pytorch/issues/63079 but this is for test_nadam under test_optim instead.

I've done some initial investigations on other nodes (8xT4,4xV100,4xA100) and other CUDA &/ PyTorch versions. I'll try to collect what worked and what didn't in separate comments. If I remember correctly it is only for combination PyTorch-1.12.x CUDA-11.7.0 on A40s I've observed the test failure.

## To Reproduce
1. Build PyTorch 1.12.1 from source with GCC 11.3.0 with CUDA 11.7.0
2. Run `python test_optim.py -k test_nadam`
ensuing error message:
```
/dev/shm/PyTorch/1.12.1/foss-2022a-CUDA-11.7.0/pytorch-v1.12.1/build/lib.linux-x86_64-cpython-310/torch/testing/_internal/common_cuda.py:19: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  CUDA11OrLater = torch.version.cuda and LooseVersion(torch.version.cuda) >= "11.0"
/apps/Arch/software/Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
F
======================================================================
FAIL: test_nadam (__main__.TestOptim)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cephyr/NOBACKUP/priv/c3-staff/vikren/build/EasyBuild/PyTorch/test_optim.py", line 671, in test_nadam
    self._test_basic_cases(
  File "/cephyr/NOBACKUP/priv/c3-staff/vikren/build/EasyBuild/PyTorch/test_optim.py", line 259, in _test_basic_cases
    self._test_state_dict(
  File "/cephyr/NOBACKUP/priv/c3-staff/vikren/build/EasyBuild/PyTorch/test_optim.py", line 241, in _test_state_dict
    self.assertEqual(bias, bias_cuda)
  File "/dev/shm/PyTorch/1.12.1/foss-2022a-CUDA-11.7.0/pytorch-v1.12.1/build/lib.linux-x86_64-cpython-310/torch/testing/_internal/common_utils.py", line 2219, in assertEqual
    assert_equal(
  File "/dev/shm/PyTorch/1.12.1/foss-2022a-CUDA-11.7.0/pytorch-v1.12.1/build/lib.linux-x86_64-cpython-310/torch/testing/_comparison.py", line 1095, in assert_equal
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 10 (10.0%)
Greatest absolute difference: 1.609325408935547e-05 at index (1,) (up to 1e-05 allowed)
Greatest relative difference: 1.477008233390933e-05 at index (1,) (up to 1.3e-06 allowed)

----------------------------------------------------------------------
Ran 1 test in 5.343s

FAILED (failures=1)
```

### Versions

```
Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Rocky Linux 8.6 (Green Obsidian) (x86_64)
GCC version: (GCC) 11.3.0
Clang version: Could not collect
CMake version: version 3.23.1
Libc version: glibc-2.28

Python version: 3.10.4 (main, Aug 14 2022, 22:57:54) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration: 
GPU 0: NVIDIA A40
GPU 1: NVIDIA A40
GPU 2: NVIDIA A40
GPU 3: NVIDIA A40

Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[conda] Could not collect
```

cc @vincentqb @jbschlosser @albanD @janeyx99