Repository: pytorch
Issue: Report issue for torch.nn.Linear when forwarding a 3-dim tensor.
Labels: module: numerical-stability,triaged

### 🐛 Describe the bug

Dear all,

We seemly found a bug in nn.linear forwarding, here is a minimal example:
```python

#  import
import torch
import time

# Set input size, output size, and batch size
input_size = 1024
output_size = 512
feature_size = 100
batch_size = 2

# Create test input data
x = torch.randn(batch_size, feature_size, input_size, device="cuda")

# Test function
def test_precision(dtype):
    # Convert the linear layer to the specified precision
    linear.to(dtype)
    x_dtype = x.to(dtype)

    # Perform a warm-up computation to ensure GPU is ready
    _ = linear(x_dtype)

    # Start timing
    torch.cuda.synchronize()
    start = time.time()

    # Perform forward pass
    a1 = linear(x_dtype)[:1]
    a2 = linear(x_dtype[:1])

    # Calculate and display the difference
    diff = (a1 - a2).abs().sum()
    print(f"Diff error: {diff} at {dtype}")

    # End timing
    torch.cuda.synchronize()
    end = time.time()

    # Compute elapsed time
    return end - start

# Define the linear layer (without bias)
linear = torch.nn.Linear(input_size, output_size, bias=False).cuda()

print("linear \"WITHOUT\" bias")
# Test computation time for different precisions
time_bfloat16 = test_precision(torch.bfloat16)
time_float16 = test_precision(torch.float16)
time_float32 = test_precision(torch.float32)

# Output results
print(f"bfloat16 forward pass time: {time_bfloat16:.6f} seconds")
print(f"float16 forward pass time: {time_float16:.6f} seconds")
print(f"float32 forward pass time: {time_float32:.6f} seconds")

# Define the linear layer (with bias)
linear = torch.nn.Linear(input_size, output_size, bias=True).cuda()

print("linear \"WITH\" bias")
time_bfloat16 = test_precision(torch.bfloat16)
time_float16 = test_precision(torch.float16)
time_float32 = test_precision(torch.float32)

# Output results
print(f"bfloat16 forward pass time: {time_bfloat16:.6f} seconds")
print(f"float16 forward pass time: {time_float16:.6f} seconds")
print(f"float32 forward pass time: {time_float32:.6f} seconds")
```

Definitely, the ```diff```  variable  in the code should be zero mathematically, however, in our test case the output is

```
linear "WITHOUT" bias
Diff error: 38.25 at torch.bfloat16
Diff error: 4.76953125 at torch.float16
Diff error: 0.0046875495463609695 at torch.float32
bfloat16 forward pass time: 0.037810 seconds
float16 forward pass time: 0.000848 seconds
float32 forward pass time: 0.000678 seconds
linear "WITH" bias
Diff error: 0.0 at torch.bfloat16
Diff error: 6.48828125 at torch.float16
Diff error: 0.004732653498649597 at torch.float32
bfloat16 forward pass time: 0.000422 seconds
float16 forward pass time: 0.000555 seconds
float32 forward pass time: 0.000377 seconds
``` 

Especially, if we use ```bfloat16```, the ```diff``` will be huge, we believe something may goes wrong here.

# My question is:

Question 1: In linear tensor computation, according to the mathematical formula, computations for different samples within the same batch should be independent. However, the results in our code indicate that when processing multiple samples in a batch, the results differ from calculating each sample individually. Is this normal?

Question 2: Is this due to numerical stability issues with floating-point numbers?

### Versions

[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.1.3
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] torch==2.5.1
[pip3] triton==3.1.0
[conda] cuda-cudart               12.6.77              h5888daf_0    conda-forge
[conda] cuda-cudart-dev           12.6.77              h5888daf_0    conda-forge
[conda] cuda-cudart-dev_linux-64  12.6.77              h3f2d84a_0    conda-forge
[conda] cuda-cudart-static        12.6.77              h5888daf_0    conda-forge
[conda] cuda-cudart-static_linux-64 12.6.77              h3f2d84a_0    conda-forge
[conda] cuda-cudart_linux-64      12.6.77              h3f2d84a_0    conda-forge
[conda] cuda-cupti                12.6.80              hbd13f7d_0    conda-forge
[conda] cuda-libraries            12.6.2               ha770c72_0    conda-forge
[conda] cuda-libraries-dev        12.6.2               ha770c72_0    conda-forge
[conda] cuda-libraries-static     12.6.2               ha770c72_0    conda-forge
[conda] cuda-nvrtc                12.6.77              hbd13f7d_0    conda-forge
[conda] cuda-nvrtc-dev            12.6.77              h5888daf_0    conda-forge
[conda] cuda-nvrtc-static         12.6.77              h5888daf_0    conda-forge
[conda] cuda-nvtx                 12.6.77              hbd13f7d_0    conda-forge
[conda] cuda-opencl               12.6.77              hbd13f7d_0    conda-forge
[conda] cuda-opencl-dev           12.6.77              h5888daf_0    conda-forge
[conda] libcublas                 12.6.3.3             hbd13f7d_1    conda-forge
[conda] libcublas-dev             12.6.3.3             h5888daf_1    conda-forge
[conda] libcublas-static          12.6.3.3             h5888daf_1    conda-forge
[conda] libcufft                  11.3.0.4             hbd13f7d_0    conda-forge
[conda] libcufft-dev              11.3.0.4             h5888daf_0    conda-forge
[conda] libcufft-static           11.3.0.4             h5888daf_0    conda-forge
[conda] libcurand                 10.3.7.77            hbd13f7d_0    conda-forge
[conda] libcurand-dev             10.3.7.77            h5888daf_0    conda-forge
[conda] libcurand-static          10.3.7.77            h5888daf_0    conda-forge
[conda] libcusolver               11.7.1.2             hbd13f7d_0    conda-forge
[conda] libcusolver-dev           11.7.1.2             h5888daf_0    conda-forge
[conda] libcusolver-static        11.7.1.2             h5888daf_0    conda-forge
[conda] libcusparse               12.5.4.2             hbd13f7d_0    conda-forge
[conda] libcusparse-dev           12.5.4.2             h5888daf_0    conda-forge
[conda] libcusparse-static        12.5.4.2             h5888daf_0    conda-forge
[conda] libnvjitlink              12.6.77              hbd13f7d_1    conda-forge
[conda] libnvjitlink-dev          12.6.77              h5888daf_1    conda-forge
[conda] libnvjitlink-static       12.6.77              h5888daf_1    conda-forge
[conda] numpy                     2.1.3                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi