Repository: pytorch
Issue: No reproducibility after ONNX export of fully converted QAT model
Labels: module: onnx,oncall: quantization,triaged

### 🐛 Describe the bug

When I use quantization-aware training, results are not reproducible between:
1. fake quantized model
2.  real quantized model
3. exported ONNX model

### Code example
```python
import torch
import onnxruntime as ort

torch.manual_seed(42)

def dummy_training(model):
    model.train()
    image = torch.rand(1, 3, 224, 224)
    model(image)
    # ...

class DummyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.quant = torch.ao.quantization.QuantStub()
        self.conv = torch.nn.Conv2d(3, 1, 1)
        self.bn = torch.nn.BatchNorm2d(1)
        self.relu = torch.nn.ReLU()
        self.dequant = torch.ao.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        x = self.dequant(x)
        return x

    def fuse_model(self):
        torch.ao.quantization.fuse_modules(self, ['conv', 'bn', 'relu'], inplace=True)

model_fp32 = DummyModel()

# prepare qat
model_fp32.eval()
model_fp32.qconfig = torch.ao.quantization.get_default_qat_qconfig('x86')
model_fp32.fuse_model()
model_fp32.train()
model_fake_quant = torch.ao.quantization.prepare_qat(model_fp32)

# run training
dummy_training(model_fake_quant)

# quantize
model_fake_quant.eval()
model_fake_quant.apply(torch.ao.quantization.disable_observer)
model_fake_quant.apply(torch.nn.intrinsic.qat.freeze_bn_stats)
model_real_quant = torch.ao.quantization.convert(model_fake_quant)

# create onnx model
torch.onnx.export(model_real_quant, torch.rand(1, 3, 224, 224), "quantized.onnx", input_names=["input"], output_names=["output"])
model_onnx = ort.InferenceSession("quantized.onnx", providers=["CPUExecutionProvider"])

# test reproducability
x = torch.rand(1, 3, 224, 224)
res_fake_quant = model_fake_quant(x)
res_real_quant = model_real_quant(x)
res_onnx = model_onnx.run(None, {"input": x.numpy()})[0]

# all asserts will fail!!!
# torch.testing.assert_close(res_fake_quant, res_real_quant)
torch.testing.assert_close(res_real_quant, torch.tensor(res_onnx))
# torch.testing.assert_close(res_fake_quant, torch.tensor(res_onnx))
```

### Error
```
Traceback (most recent call last):
  File "D:\Projekte\Quantization 2.0\qat_reproduce3.py", line 62, in <module>
    torch.testing.assert_close(res_real_quant, torch.tensor(res_onnx))
  File "D:\Tools\Venv\Pytorch_2.5\lib\site-packages\torch\testing\_comparison.py", line 1524, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

Mismatched elements: 49 / 50176 (0.1%)
Greatest absolute difference: 0.011374831199645996 at index (0, 0, 105, 1) (up to 1e-05 allowed)
Greatest relative difference: 0.021739039570093155 at index (0, 0, 116, 173) (up to 1.3e-06 allowed)
```

### Expected Behavior

According to https://discuss.pytorch.org/t/173684 we cannot expect a fake quantized model to behave the same as the quantized model. However, I would expect that once the model is fully quantized, it behaves the same between Pytorch and ONNX Runtime. This was the case for all float32 models I tested.

In this minimal example, the problem ist not severe, but for a real model (ResNet18), the number of mismatched elements grows to over 40% and the greatest absolute difference to 0.05 (greated rel diff to infinity).

### Versions

```
Collecting environment information...
PyTorch version: 2.4.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro (10.0.19045 64-Bit)
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.6.55
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: Quadro RTX 5000
GPU 1: Quadro RTX 5000

Nvidia driver version: Could not collect
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Name: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Manufacturer: GenuineIntel
Family: 179
Architecture: 9
ProcessorType: 3
DeviceID: CPU0
CurrentClockSpeed: 2201
MaxClockSpeed: 2201
L2CacheSize: 3072
L2CacheSpeed: None
Revision: 20225

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] onnx==1.17.0
[pip3] onnxruntime-gpu==1.19.2
[pip3] torch==2.4.1+cu118
[pip3] torchvision==0.19.1+cu118
[conda] Could not collect
```

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @msaroufim