Repository: pytorch
Issue: Eager PTDQ Performs Worse Than Non-Quantized Linear Layer on CPU(in Terms of Speed)
Labels: triaged

### 🐛 Describe the bug

While Dynamically post-training quantized the following linear model:
```python
class Net(nn.Module):
    def __init__(self):
      super().__init__()
      self.layer = nn.Linear(100, 20000*3)
    def forward(self, x):
        return self.layer(x)
```
It performed worse than the non-quantized model while inferring.

Code To reproduce:
```python
import torch
import torch.nn as nn
import time
N = 200


class Net(nn.Module):
    def __init__(self):
      super().__init__()
      self.layer = nn.Linear(100, 20000*3)
    def forward(self, x):
        return self.layer(x)


def time_model(model):
  start_time = time.time()
  x = torch.rand(1, N, 100)
  model(x)
  print(f"Model Running time {time.time() - start_time}")
    
def main():
    model = Net()
    print("regular model")
    print(model)
    time_model(model)
    model_int8 = torch.ao.quantization.quantize_dynamic(
    model,  
    {torch.nn.Linear},
    dtype=torch.qint8)
    print("int8 model")
    print(model_int8)
    time_model(model_int8)
main()
```
If the variable N is changed to 1 the quantized model perform better, as expected.

Output:
```
regular model
Net(
  (layer): Linear(in_features=100, out_features=60000, bias=True)
)
Model Running time 0.07307147979736328
int8 model
Net(
  (layer): DynamicQuantizedLinear(in_features=100, out_features=60000, dtype=torch.qint8, qscheme=torch.per_tensor_affine)
)
Model Running time 0.10798835754394531
```
Profiler Output (On different run)
# The non quantisized run:
                  Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  Source Location                                              
          aten::linear         0.22%      72.000us        99.60%      32.807ms      32.807ms             1  <built-in function linear>                                   
                                                                                                            torch/nn/modules/linear.py(113): forward                     
                                                                                                            nn.Module: Linear_0                                          
                                                                                                            ./test.py(11): forward                                       
                                                                                                            nn.Module: Net_0    
# The quantisized run:
                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  Source Location                                                             
    quantized::linear_dynamic        99.59%      42.069ms        99.69%      42.113ms      42.113ms             1  <built-in method linear_dynamic of PyCapsule object at 0x7f1490949020>      
                                                                                                                   torch/_ops.py(497): __call__                                                
                                                                                                                   torch/ao/nn/quantized/dynamic/modules/linear.py(47): forward                
                                                                                                                   nn.Module: Linear_0                                                         
                                                                                                                   ./test.py(11): forward  
```


### Versions

PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.25.2
Libc version: glibc-2.31

Python version: 3.10.12 (main, Jun  7 2023, 12:45:35) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.107+-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              2
Core(s) per socket:              1
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           79
Model name:                      Intel(R) Xeon(R) CPU @ 2.20GHz
Stepping:                        0
CPU MHz:                         2200.218
BogoMIPS:                        4400.43
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       32 KiB
L1i cache:                       32 KiB
L2 cache:                        256 KiB
L3 cache:                        55 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable; SMT Host state unknown
Vulnerability Meltdown:          Vulnerable
Vulnerability Mmio stale data:   Vulnerable
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Vulnerable
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==2.0.1+cu118
[pip3] torchaudio==2.0.2+cu118
[pip3] torchdata==0.6.1
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.15.2
[pip3] torchvision==0.15.2+cu118
[pip3] triton==2.0.0
[conda] Could not collect