Repository: pytorch
Issue: Inconsistent Bias Broadcasting Results in PyTorch GPU `F.linear(...)` and `F.conv2d(...)` Operations
Labels: needs reproduction,module: cudnn,module: cuda,triaged

The following code snippet demonstrates an issue with `torch.nn.functional.linear(...)` and `torch.nn.functional.conv2d(...)` when using automatic broadcasting on the GPU. Specifically, the outputs of two mathematically equivalent operations diverge when broadcasting is involved. While only `F.linear(...)` and `F.conv2d(...)` have been tested, this suggests that other operations utilizing broadcasting might exhibit similar inconsistencies.

```python
import torch
import torch.nn.functional as F

# Initialize weights, input, and bias on GPU
W = torch.randn(8192, 8192, device='cuda')
x = torch.randn(1, 8192, device='cuda')
bias = torch.randn(8192, device='cuda')

# Perform linear operation with bias
y1 = F.linear(x, W, bias)

# Perform linear operation without bias and add bias manually
y2 = F.linear(x, W, bias=None) + bias

assert torch.allclose(y1, y2), \
    'Outputs are not equal! Max error: {}'.format(torch.max(torch.abs(y1 - y2)))
```
On the GPU (e.g., NVIDIA A10, CUDA 12.0), the assertion fails with a maximum error around 0.0018. On the CPU, `torch.allclose(...)` passes consistently.


## Key Observations

- **GPU-Specific Issue**: The discrepancy occurs exclusively on GPU hardware.
- **Bias Shape Sensitivity**:
  - Affected: When the bias is a 1D tensor (`bias.shape == (8192,)`), requiring automatic broadcasting.
  - Unaffected: Reshaping the bias to match the input shape (e.g., `bias = torch.randn(1, 8192, device='cuda')`) eliminates the issue. 
- **Non-Deterministic Errors**: The magnitude of the error varies unpredictably despite adhering to PyTorch’s [reproducibility guide](https://pytorch.org/docs/stable/notes/randomness.html).
- **Potential for Broader Impact**: Since broadcasting is a common operation in many PyTorch functions, similar discrepancies might arise in other GPU-accelerated scenarios beyond those mentioned in this issue regarding `F.linear(...)` and `F.conv2d(...)`.


## Workaround

To circumvent the issue, ensure that the bias tensor has a shape compatible with the input tensor, thereby avoiding automatic broadcasting:
```python
bias = torch.randn(1, 8192, device='cuda')
```
This adjustment ensures consistent and accurate results on both GPU and CPU.


## Environment Details

Configurations applied to enforce reproducibility (issue persists despite these settings):
```python
import os
os.environ["CUBLAS_WORKSPACE_CONFIG"]=':4096:8'

import torch
torch.backends.cudnn.enabled = True
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.allow_tf32 = False
```


### Versions

PyTorch version: 2.2.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10
Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 57 bits virtual
CPU(s):                          30
On-line CPU(s) list:             0-29
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       30
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           106
Model name:                      Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
Stepping:                        6
CPU MHz:                         2593.970
BogoMIPS:                        5187.94
Virtualization:                  VT-x
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       960 KiB
L1i cache:                       960 KiB
L2 cache:                        120 MiB
L3 cache:                        480 MiB
NUMA node0 CPU(s):               0-29
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid fsrm md_clear arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==8.9.2.26
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-nccl-cu12==2.19.3
[pip3] nvidia-nvjitlink-cu12==12.6.77
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] torch==2.2.2+cu121
[pip3] torchaudio==2.2.2+cu121
[pip3] torchvision==0.17.2+cu121
[pip3] triton==2.2.0
[conda] Could not collect

cc @csarofeen @ptrblck @xwang233 @eqy @msaroufim