Repository: pytorch
Issue: [inductor][cpu]mobilenet_v2_quantized_qat float32 single thread static/dynamic shape CPP/default wrapper performance regression in 2024-04-28 nightly release
Labels: triaged,oncall: pt2,module: inductor

### 🐛 Describe the bug

<p>float32 static shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>mobilenet_v2_quantized_qat</td>
      <td>single</td>
      <td>1</td>
      <td>0.986425</td>
      <td>0.006968785</td>
      <td>0.006874183743625</td>
      <td>0.096932</td>
      <td>1</td>
      <td>0.990124</td>
      <td>0.006050362</td>
      <td>0.005990608624888</td>
      <td>0.093676</td>
      <td>1.0</td>
      <td>0.87</td>
      <td>0.87</td>
      <td>0.97</td>
    </tr>
  </tbody>

</table>

<p>float32 static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>mobilenet_v2_quantized_qat</td>
      <td>single</td>
      <td>1</td>
      <td>0.985836</td>
      <td>0.006905878999999999</td>
      <td>0.006808064129843999</td>
      <td>0.096427</td>
      <td>1</td>
      <td>0.991287</td>
      <td>0.006014836</td>
      <td>0.005962428733932</td>
      <td>0.093884</td>
      <td>0.99</td>
      <td>0.88</td>
      <td>0.87</td>
      <td>0.97</td>
    </tr>
  </tbody>

</table>

<p>float32 dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>mobilenet_v2_quantized_qat</td>
      <td>single</td>
      <td>1</td>
      <td>0.985912</td>
      <td>0.006797219</td>
      <td>0.006701459778728</td>
      <td>0.096001</td>
      <td>1</td>
      <td>0.988582</td>
      <td>0.006000043</td>
      <td>0.005931534509026</td>
      <td>0.093643</td>
      <td>1.0</td>
      <td>0.89</td>
      <td>0.88</td>
      <td>0.98</td>
    </tr>
  </tbody>


</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>d6015d42</td>
      <td>main</td>
      <td>d6015d42</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>7478b7f1cac9686f00edf3db4667cf86d2421531</td>
      <td>main</td>
      <td>bad8d25881d850eaf0b326f6ce5c78305e38c001</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+2c4665f</td>
      <td>main</td>
      <td>0.19.0a0+2c4665f</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.2.0a0+ea437b3</td>
      <td>main</td>
      <td>2.2.0a0+ea437b3</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh single inference performance torchbench mobilenet_v2_quantized_qat float32 first dynamic/static default/cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/e62169a8fa84246babe98410bb5600693db62a14
[torchbench-mobilenet_v2_quantized_qat-inference-float32-dynamic-cpp-single-performance-drop_guilty_commit.log](https://github.com/pytorch/pytorch/files/15233790/torchbench-mobilenet_v2_quantized_qat-inference-float32-dynamic-cpp-single-performance-drop_guilty_commit.log)
cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @WeizhuoZhang-intel @chuanqi129