Repository: pytorch
Issue: Tensor's storage changes computation outcome for CPU tensors.
Labels: module: cpu,triaged,matrix multiplication

### 🐛 Describe the bug

## High level

Tensor's storage changes the outcome of computation based on it. That is, an identical tensor residing in shared memory will yield a different `matmul` result from a counterpart residing in local memory. 

Assuming that `matmul` computation is deterministic for CPU for a given hardware/software computation that means that one of the two computations is erroneous.

## Detailed bug description

This seems to occur any time the tensor is relocated from local memory to shared memory and then a copy of the shared memory is created. Originally the bug was found while debugging multiprocessing message passing between processes, but the following is the most bare-bone reproduction of the issue:

```python
import torch
import torch.nn as nn

IN_DIM = 400 

if __name__ == '__main__':
    linear = nn.Linear(IN_DIM, 100)
    linear.weight.share_memory_()

    linear_2 = nn.Linear(IN_DIM, 100) 
    with torch.no_grad():
        linear_2.weight = linear.weight
        linear_2.bias = linear.bias
        linear.weight = nn.Parameter(linear_2.weight.clone().detach())

    input = torch.randn((1, IN_DIM))
    assert torch.equal(linear.weight, linear_2.weight) 
    assert torch.equal(linear.bias, linear_2.bias) 
    assert torch.equal(linear(input), linear_2(input))
```

A run will result  in:

```
% python -m repro
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/private/tmp/repro.py", line 19, in <module>
    assert torch.equal(linear(input), linear_2(input))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
```

Despite the assertions around `weight` and `bias` passing, the computation based on them will produce results which are not identical. A few things to note:

1. The bug only reproduces for "large enough" `IN_DIM` which differs by `OUT_DIM` of the linear layer. This makes me think that the issue is in memory alignment or other related problems.
2. The bug only reproduces if exactly one tensor resides in shared memory. It does not reproduce if both tensors are in shared memory or neither tensor is in shared memory (the computations are equivalent holding the storage invariant, but differ if the storage differs)

Related code:
1. Storage copy (https://github.com/pytorch/pytorch/blob/1ac402a96ca54e2d882b6daae4bf9e8c956ae03e/aten/src/ATen/StorageUtils.cpp#L35-L54)
2. Matmul (https://github.com/pytorch/pytorch/blob/ed120b08c4828c39f116cfe1fb39195c844be485/aten/src/ATen/native/LinearAlgebra.cpp#L1402)

## Root cause

I have not yet been able to pin-point the exact lines of code involved. I assume that PyTorch Devs will have an easier time doing so given environment setup. Happy to dig deeper as needed.

### Versions

Collecting environment information...
PyTorch version: 2.2.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.4.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.29.2
Libc version: N/A

Python version: 3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 15.0.0 (clang-1500.3.9.4)] (64-bit runtime)
Python platform: macOS-14.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Max

Versions of relevant libraries:
[pip3] mypy==1.9.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] torch==2.2.1
[conda] Could not collect


cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10