Repository: pytorch
Issue: cpu - gpu calculation results differs by far with torch.nn.functional.linear
Labels: module: numerical-stability,module: cuda,triaged

### 🐛 Describe the bug

I tried two `torch.nn.functional.linear` with the same input, one on cpu and another on gpu.
The gap between the two seems to be too big. I have seen issues([1](https://github.com/pytorch/pytorch/issues/23392), [2](https://discuss.pytorch.org/t/significance-of-the-difference-between-cpu-and-gpu-results/14739), [3](https://discuss.pytorch.org/t/why-different-results-when-multiplying-in-cpu-than-in-gpu/1356/3))  reporting the small calculation difference, which makes sense to some extent, but I am having a hard time getting my head around what could cause the big difference as below. 
```python
torch.nn.functional.linear(hidden_states.cpu(), w.cpu(), b.cpu()).norm()
# tensor(182.7500, dtype=torch.float16)
torch.nn.functional.linear(hidden_states, w, b).norm()
# tensor(233.7366, device='cuda:0')
```
Here are some additional info.
```python
type(hidden_states)  #<class 'torch.Tensor'>
hidden_states.dtype  # torch.float16
hidden_states.size()  #torch.Size([23, 2, 2048])
type(w)  # <class 'torch.Tensor'>
w.dtype  # torch.float16
w.size()  #torch.Size([2048, 2048])
type(b)  # <class 'torch.Tensor'>
b.dtype  # torch.float16
b.size()  #torch.Size([2048])
```


### Versions

PyTorch version: 1.11.0a0+b6df043
Is debug build: False
CUDA used to build PyTorch: 11.5
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.21.3
Libc version: glibc-2.31

Python version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-4.19.93-1.nbp.el7.x86_64-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.5.50
GPU models and configuration: GPU 0: A100-SXM-80GB
Nvidia driver version: 460.73.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.4
[pip3] nvidia-dlprof-pytorch-nvtx==1.7.0
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.11.0a0+b6df043
[pip3] torch-tensorrt==1.0.0a0
[pip3] torchtext==0.12.0a0
[pip3] torchvision==0.11.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.5                      281    conda-forge
[conda] mkl-include               2019.5                      281    conda-forge
[conda] numpy                     1.21.4           py38he2449b9_0    conda-forge
[conda] nvidia-dlprof-pytorch-nvtx 1.7.0                    pypi_0    pypi
[conda] pytorch-quantization      2.1.2                    pypi_0    pypi
[conda] torch                     1.11.0a0+b6df043          pypi_0    pypi
[conda] torch-tensorrt            1.0.0a0                  pypi_0    pypi
[conda] torchtext                 0.12.0a0                 pypi_0    pypi
[conda] torchvision               0.11.0a0                 pypi_0    pypi

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @ngimel @zasdfgbnm @ptrblck