Repository: pytorch
Issue: PyTorch 1.9.0, test_optim fails on Nvidia A10.
Labels: module: optimizer,module: cuda,module: tests,triaged

## 🐛 Bug

On Nvidia A10 cards, after building PyTorch from source, test_optim fails.

## To Reproduce

Steps to reproduce the behavior:

1. Build PyTorch 1.9.0 from source with GCC 10.2.0, NCCL 2.8.3, and CUDA 11.1.1 on a node with Nvidia A10 graphics hardware
2. Observe test_optim fail as per above.


````
======================================================================
FAIL: test_adadelta (__main__.TestOptim)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/eb-mnzps9fx/tmpuyuyq4xt/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 439, in wrapper
    fn(*args, **kwargs)
  File "test_optim.py", line 465, in test_adadelta
    self._test_basic_cases(
  File "test_optim.py", line 204, in _test_basic_cases
    self._test_state_dict(
  File "test_optim.py", line 192, in _test_state_dict
    self.assertEqual(weight, weight_cuda)
  File "/tmp/eb-mnzps9fx/tmpuyuyq4xt/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 1355, in assertEqual
    super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
AssertionError: False is not true : Tensors failed to compare as equal!With rtol=1.3e-06 and atol=1e-05, found 5 element(s) (out of 50) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 2.002716064453125e-05 (-0.7925083637237549 vs. -0.7925283908843994), which occurred at index (9, 1).

----------------------------------------------------------------------
Ran 105 tests in 36.167s

FAILED (failures=1)
test_optim failed!
````

## Expected behavior

Tests not failing.

## Environment

Gathered after disabling the test in question:

````
Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux release 8.4 (Ootpa) (x86_64)
GCC version: (GCC) 10.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.8.6 (default, Mar 16 2021, 10:03:15)  [GCC 10.2.0] (64-bit runtime)
Python platform: Linux-4.18.0-305.10.2.el8_4.x86_64-x86_64-with-glibc2.2.5
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration:
GPU 0: NVIDIA A10
GPU 1: NVIDIA A10
GPU 2: NVIDIA A10
GPU 3: NVIDIA A10

Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] torch==1.9.0
[conda] Could not collect
````

 - PyTorch Version (e.g., 1.0): 1.9.0
 - OS (e.g., Linux): Linux RHEL 8.4
 - How you installed PyTorch (`conda`, `pip`, source): Source 
 - Build command you used (if compiling from source): USE_CUPTI_SO=1 PYTORCH_BUILD_VERSION=1.9.0 PYTORCH_BUILD_NUMBER=1 MAX_JOBS=64 BLAS=MKL INTEL_MKL_DIR=$MKLROOT USE_FFMPEG=1 BU
ILD_CUSTOM_PROTOBUF=0 USE_IBVERBS=1 USE_CUDA=1 CUDNN_LIB_DIR=/opt/uio/modules/rhel8/easybuild/software/cuDNN/8.0.4.30-CUDA-11.1.1/lib64 CUDNN_INCLUDE_DIR=/opt/uio/modules/rhe
l8/easybuild/software/cuDNN/8.0.4.30-CUDA-11.1.1/include USE_SYSTEM_NCCL=1 NCCL_INCLUDE_DIR=/opt/uio/modules/rhel8/easybuild/software/NCCL/2.8.3-GCCcore-10.2.0-CUDA-11.1.1/in
clude USE_METAL=0   /opt/uio/modules/rhel8/easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build
 - Python version: 3.8.6
 - CUDA/cuDNN version: 11.11.1 / 8.0.4.30
 - GPU models and configuration: 4xA10, all local

## Additional context

Tests seem fine on other Nvidia hardware. The failure occurs with or without IMKL support on the same hardware.


cc @vincentqb @iramazanli @ngimel @mruberry @VitalyFedyunin @walterddr