Repository: pytorch
Issue: Large numerical error when applying nn.Linear in RTX A6000 with cuda>=11.1
Labels: high priority,module: cuda,triaged,module: tf32

### 🐛 Describe the bug

# Applying a simple Linear layer (Y = aX) is generating incorrect output with A6000 GPUs using cuda >=11.1

## Report: 
1.In the code snippet, we have a simple linear layer (without bias). First we use nn.Linear to compute Y=aX. Next we explicitly multiply the two tensors, We expect the norm of the difference to be 0 in all cases (can be confirmed by uncommenting the line setting device to "cpu"). However a significant error is seen with A6000 GPUs using cuda >= 11.1 (Please see output table).
2. No error observed for CPU, and other GPUs at our disposal (NVIDIA RTX, TITAN X )
3. With A6000, **no error for cuda 11.0**.
4. With A6000, **error confirmed for cuda 11.1 and  cuda 11.3**. 
5. This error is deadly serious since it is an insidious corruption, without any obvious outward symptoms. In fact, if we uncomment the other line of code, it can be seen that a significant error is introduced even when multiplying with 1.
 

## Code Snippet:
```python
import torch
d = "cuda:0"
#d = "cpu"
x = [pow(10,x) for x in list(range(10))]
for size in x:
    data = torch.rand(size,1)
    lin = torch.nn.Linear(1,1,bias=False).to(d)
    #lin.weight.data = torch.tensor([[1.0]])
    print (size,torch.norm(data.to(d)*lin.weight.to(d) - lin.to(d)(data.to(d))).item())
```
## Output
| size | Error Norm | 
| ------------------ | ---------------------------------------|  
| 1 |  5.2422285079956055e-05 |
| 10 | 0.0003333788481540978 | 
| 100 | 0.0008036142098717391 | 
| 1000 | 2.7209020117879845e-05 | 
| 10000 | 0.0 | 
| 100000 | 0.005546107422560453 | 
| 1000000 | 0.13998229801654816 | 
| 10000000 | 0.05942477285861969 | 
| 100000000 | 1.474966287612915 | 
| 1000000000 | 2.536698341369629 | 






### Versions

Collecting environment information...
PyTorch version: 1.10.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-94-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000

Nvidia driver version: 470.86
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.0
[pip3] torch==1.10.1+cu113
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==2.0.3
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[pip3] torch-spline-conv==1.2.1
[pip3] torchaudio==0.10.1+cu113
[pip3] torchvision==0.11.2+cu113
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.2.0           h06a4308_296
[conda] mkl-service               2.3.0            py38h27cfd23_1
[conda] mkl_fft                   1.3.0            py38h42c9631_2
[conda] mkl_random                1.2.1            py38ha9443f7_2
[conda] mypy_extensions           0.4.3                    py38_0
[conda] numpy                     1.20.1           py38h93e21f0_0
[conda] numpy-base                1.20.1           py38h7d8b39e_0
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1


cc @ezyang @gchanan @zou3519 @ngimel @zasdfgbnm @ptrblck