Repository: pytorch
Issue: [inductor][cpu]speech_transformer AMP single/multiple thread static/dynamic shape CPP/default wrapper performance regression in 2024-05-12 nightly release
Labels: triaged,oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>AMP static shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>multiple</td>
      <td>1</td>
      <td>0.958469</td>
      <td>0.030251814</td>
      <td>0.028995425912766</td>
      <td>69.833969</td>
      <td>1.0</td>
      <td>1.263245</td>
      <td>0.023321511</td>
      <td>0.029460782163194997</td>
      <td>34.216882</td>
      <td>0.76</td>
      <td>1.02</td>
      <td>0.77</td>
      <td>0.49</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>single</td>
      <td>1</td>
      <td>0.996765</td>
      <td>0.217230587</td>
      <td>0.21652784605105502</td>
      <td>69.387519</td>
      <td>1.0</td>
      <td>1.268162</td>
      <td>0.176525437</td>
      <td>0.223862851236794</td>
      <td>33.153806</td>
      <td>0.79</td>
      <td>1.03</td>
      <td>0.81</td>
      <td>0.48</td>
    </tr>
  </tbody>

</table>

<p>AMP dyanmic shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>multiple</td>
      <td>1</td>
      <td>0.951825</td>
      <td>0.030079704</td>
      <td>0.0286306142598</td>
      <td>69.78338</td>
      <td>1.0</td>
      <td>1.258736</td>
      <td>0.023420975</td>
      <td>0.029480824387600003</td>
      <td>34.077379</td>
      <td>0.76</td>
      <td>1.03</td>
      <td>0.78</td>
      <td>0.49</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>single</td>
      <td>1</td>
      <td>0.996724</td>
      <td>0.218491288</td>
      <td>0.21777551054051203</td>
      <td>69.378941</td>
      <td>1.0</td>
      <td>1.241771</td>
      <td>0.178057441</td>
      <td>0.221106566568011</td>
      <td>33.172262</td>
      <td>0.8</td>
      <td>1.02</td>
      <td>0.81</td>
      <td>0.48</td>
    </tr>
  </tbody>

</table>

<p>AMP static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>multiple</td>
      <td>1</td>
      <td>0.982142</td>
      <td>0.029919203</td>
      <td>0.029384905872826</td>
      <td>207.79774</td>
      <td>1.0</td>
      <td>1.331547</td>
      <td>0.022101082</td>
      <td>0.029428629433854003</td>
      <td>75.792615</td>
      <td>0.74</td>
      <td>1.0</td>
      <td>0.74</td>
      <td>0.36</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>single</td>
      <td>1</td>
      <td>0.997013</td>
      <td>0.21735344</td>
      <td>0.21670420527472</td>
      <td>207.207765</td>
      <td>1.0</td>
      <td>1.273872</td>
      <td>0.17650107299999998</td>
      <td>0.22483977486465595</td>
      <td>75.687333</td>
      <td>0.78</td>
      <td>1.04</td>
      <td>0.81</td>
      <td>0.37</td>
    </tr>
  </tbody>

</table>

<p>AMP dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>multiple</td>
      <td>1</td>
      <td>0.972185</td>
      <td>0.029728354</td>
      <td>0.028901459833489997</td>
      <td>208.31205</td>
      <td>1.0</td>
      <td>1.347673</td>
      <td>0.021702161</td>
      <td>0.029247416421353</td>
      <td>75.875923</td>
      <td>0.72</td>
      <td>1.01</td>
      <td>0.73</td>
      <td>0.36</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>speech_transformer</td>
      <td>single</td>
      <td>1</td>
      <td>0.997032</td>
      <td>0.218706831</td>
      <td>0.218057709125592</td>
      <td>207.342141</td>
      <td>1.0</td>
      <td>1.278801</td>
      <td>0.175010154</td>
      <td>0.22380315994535402</td>
      <td>75.659505</td>
      <td>0.78</td>
      <td>1.03</td>
      <td>0.8</td>
      <td>0.36</td>
    </tr>
  </tbody>


</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>d6015d42</td>
      <td>main</td>
      <td>d6015d42</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>02093b6c6ae1046368e2500881d0bb5880873386</td>
      <td>main</td>
      <td>6d30803d64953955df63da56833bf4eb52249aae</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+06ad737</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.2.0a0+ea437b3</td>
      <td>main</td>
      <td>2.2.0a0+ea437b3</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple/single inference performance torchbench speech_transformer amp first dynamic/static default/cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/0935b3d794c1b96911084fcc8c7748db5d52b0aa
[torchbench-speech_transformer-inference-amp-static-default-single-performance-drop_guilty_commit.log](https://github.com/pytorch/pytorch/files/15319570/torchbench-speech_transformer-inference-amp-static-default-single-performance-drop_guilty_commit.log)
cc @ezyang @chauhang @penguinwu @msaroufim @bdhirsh @anijain2305 @WeizhuoZhang-intel @chuanqi129