Repository: pytorch
Issue: Quantization error between fake-quantized model and quantized model using the new observer
Labels: oncall: quantization,low priority,triaged

### 🐛 Describe the bug

Interesting findings:
```python
from collections import Counter

import torch
import torch.nn as nn
import warnings

from distutils.version import LooseVersion

warnings.simplefilter("ignore")


USE_LEGACY_FAKE_QUANT = True


class SimpleConv(nn.Module):
    def __init__(self) -> None:
        super().__init__()

        self.q = torch.quantization.QuantStub()
        self.conv = nn.Conv2d(3, 3, 1)
        self.dq = torch.quantization.DeQuantStub()

    def forward(self, x):
        return self.dq(self.conv(self.q(x)))


x = torch.rand(1, 3, 224, 224)

m = SimpleConv()
m.train()

backend = 'qnnpack'

if USE_LEGACY_FAKE_QUANT:
    if LooseVersion(torch.__version__) >= '1.12.0':
        m.qconfig = torch.quantization.get_default_qat_qconfig(backend, 0)
    else:
        m.qconfig = torch.quantization.get_default_qat_qconfig(backend, None)
else:
    m.qconfig = torch.quantization.get_default_qat_qconfig(backend)

torch.backends.quantized.engine = backend

m = torch.quantization.prepare_qat(m, inplace=True)

for _ in range(10):
    m(x)

with torch.no_grad():
    m.eval()
    m.apply(torch.quantization.disable_observer)

    y = m(x)
    print('Result is stable during QAT:', torch.allclose(y, m(x)))

    m = torch.quantization.convert(m, inplace=True)

    y_ = m(x)
    print('Result is stable after conversion:', torch.allclose(y_, m(x)))

    print('Result is stable regardless of conversion:', torch.allclose(y, y_))

    float_diff = (y - m(x)).abs().flatten()
    int_diff = torch.round(float_diff / m.conv.scale)

    counter = Counter(int_diff.tolist())
    total = int_diff.numel()
    print('Difference value (ratio):')
    for k, v in counter.most_common():
        print(f'{k}: {v} ({v/total*100:.2f}%)')
```

With `USE_LEGACY_FAKE_QUANT=True`, I get the following result.
```
Result is stable during QAT: True
Result is stable after conversion: True
Result is stable regardless of conversion: False
Difference value (ratio):
0.0: 150431 (99.94%)
1.0: 97 (0.06%)
```
The difference is always within 1 of the quantized values, and the ratio with conversion error is lower than 0.1%.

However, when I set `USE_LEGACY_FAKE_QUANT=False` (which is the default behavior for `get_default_qat_qconfig`), I get
```
Result is stable during QAT: True
Result is stable after conversion: True
Result is stable regardless of conversion: False
Difference value (ratio):
0.0: 104794 (69.62%)
1.0: 45734 (30.38%)
```
The error rate could be something between 10-30% and there could be some cases for 2 quantized value difference.

I understand that there could be some difference for models for fake-quantized and quantized models but 30% for a layer is just too much.

### Versions

Collecting environment information...
PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6.1 (x86_64)
GCC version: Could not collect
Clang version: 12.0.0 (clang-1200.0.32.28)
CMake version: version 3.18.0
Libc version: N/A

Python version: 3.8.6 (default, Oct  8 2020, 14:06:32)  [Clang 12.0.0 (clang-1200.0.32.2)] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.13.0
[pip3] torchvision==0.14.0
[conda] Could not collect

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel