Repository: pytorch
Issue: [inductor][cpu]maml fp32/amp_fp16 performance regression in 2024-11-11 nightly release
Labels: oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>fp32 static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>maml</td>
      <td>single</td>
      <td>1</td>
      <td>1.003519</td>
      <td>0.098442143</td>
      <td>0.098788560901217</td>
      <td>79.571829</td>
      <td>1</td>
      <td>1.154072</td>
      <td>0.08604004500000001</td>
      <td>0.09929640681324</td>
      <td>78.661007</td>
      <td>0.87</td>
      <td>1.01</td>
      <td>0.87</td>
      <td>0.99</td>
    </tr>
  </tbody>

</table>

<p>fp32 dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>maml</td>
      <td>single</td>
      <td>1</td>
      <td>1.003255</td>
      <td>0.098474461</td>
      <td>0.098794995370555</td>
      <td>79.525869</td>
      <td>1</td>
      <td>1.156549</td>
      <td>0.085749273</td>
      <td>0.099173235938877</td>
      <td>78.572847</td>
      <td>0.87</td>
      <td>1.0</td>
      <td>0.87</td>
      <td>0.99</td>
    </tr>
  </tbody>

</table>

<p>amp fp16 static default</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>maml</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.015477</td>
      <td>0.1186938</td>
      <td>0.1205308239426</td>
      <td>90.049904</td>
      <td>1</td>
      <td>1.193928</td>
      <td>0.10232684</td>
      <td>0.12217087942752002</td>
      <td>90.818413</td>
      <td>0.85</td>
      <td>1.01</td>
      <td>0.86</td>
      <td>1.01</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>maml</td>
      <td>single</td>
      <td>1</td>
      <td>1.086367</td>
      <td>0.98123772</td>
      <td>1.06598427816324</td>
      <td>160.030677</td>
      <td>1</td>
      <td>8.338911</td>
      <td>0.13361106</td>
      <td>1.11417073795566</td>
      <td>161.398547</td>
      <td>0.13</td>
      <td>1.05</td>
      <td>0.14</td>
      <td>1.01</td>
    </tr>
  </tbody>

</table>

the bad commit fe4fa1df9fa981f31f85170e40a754479759267f
```
/workspace/pytorch# bash inductor_single_run.sh single inference performance torchbench maml float32 first static cpp
Testing with cpp wrapper.
Testing with inductor.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  maml
running benchmark: 100%|█████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.06it/s]
0.998x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,maml,1,0.998400,98.619239,80.780985,0.705929,46.189773,65.431142,435,18,14,3,0,0,0
```
the good commit fdfd4c50bad8b7d12416d45233ad990c45cf7ef9
```
/workspace/pytorch# bash inductor_single_run.sh single inference performance torchbench maml float32 first static cpp
Testing with cpp wrapper.
Testing with inductor.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  maml
running benchmark: 100%|█████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.42it/s]
1.157x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,maml,1,1.157074,85.184368,21.318866,0.743438,46.032486,61.918413,435,18,14,3,0,0,0
```






### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>766a5e3a</td>
      <td>main</td>
      <td>e522b45c</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>5ef33e40b3c3fd2608552d3301c7255826c0e7f6</td>
      <td>main</td>
      <td>f121eab0182f7da58b39ffb84744bdc7109817e3</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.5.0a0+fa44bda</td>
      <td>main</td>
      <td>2.5.0a0+fa44bda</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob//main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh single inference performance torchbench maml float32 first static cpp
Suspected guilty commit: fe4fa1df9fa981f31f85170e40a754479759267f
[torchbench-maml-inference-float32-static-cpp-single-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/17744695/torchbench-maml-inference-float32-static-cpp-single-performance-drop_guilty_commit.log)
cc @chauhang @penguinwu @chuanqi129