Repository: pytorch
Issue: [inductor][cpu]amp fp16 llama dynamic shape cpp wrapper performance regression in 2025-01-07 nightly release
Labels: oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>amp fp16 dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>llama</td>
      <td>multiple</td>
      <td>32</td>
      <td>2.211985</td>
      <td>0.021930711999999998</td>
      <td>0.048510405983319994</td>
      <td>39.177836</td>
      <td>32</td>
      <td>2.507979</td>
      <td>0.018847622</td>
      <td>0.04726944017593801</td>
      <td>41.306366</td>
      <td>0.88</td>
      <td>0.97</td>
      <td>0.86</td>
      <td>1.05</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>llama</td>
      <td>single</td>
      <td>1</td>
      <td>3.950647</td>
      <td>0.01318508</td>
      <td>0.05208959674676</td>
      <td>37.938252</td>
      <td>1</td>
      <td>4.542274</td>
      <td>0.011483397</td>
      <td>0.05216073562477799</td>
      <td>40.390422</td>
      <td>0.87</td>
      <td>1.0</td>
      <td>0.87</td>
      <td>1.06</td>
    </tr>
  </tbody>

</table>

the last good commit: e88d06f54eeb80669a8a97322cf55c4da0519f08
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance torchbench llama amp_fp16 first dynamic cpp
Testing with dynamic shapes.
Testing with cpp wrapper.
Testing with inductor.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  llama
running benchmark: 100%|███████████████████████████████████████████████████████████████████| 50/50 [00:03<00:00, 14.90it/s]
2.818x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,llama,32,2.817930,19.087040,33.137240,0.947519,340.724531,359.596442,531,1,0,0,0,0,0
```
the bad commit: b5b419d6276e5f0a9df623b45e9fb478f93ecc4b
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance torchbench llama amp_fp16 first dynamic cpp
running benchmark: 100%|███████████████████████████████████████████████████████████████████| 50/50 [00:03<00:00, 13.42it/s]
2.532x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,llama,32,2.532279,22.605699,37.389425,0.928050,340.260454,366.640333,531,1,0,0,0,0,0
```

### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>766a5e3a</td>
      <td>main</td>
      <td>766a5e3a</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>f2d6cfa6775601df5a038f7a4d0b37da75a53ed9</td>
      <td>main</td>
      <td>cf0b72c4ab960a847758132cc501cf793926e070</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.6.0a0+b6d4675</td>
      <td>main</td>
      <td>2.6.0a0+b6d4675</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob//main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance torchbench llama amp_fp16 first dynamic cpp
Suspected guilty commit: b5b419d6276e5f0a9df623b45e9fb478f93ecc4b
[torchbench-llama-inference-amp_fp16-dynamic-cpp-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/18432852/torchbench-llama-inference-amp_fp16-dynamic-cpp-multiple-performance-drop_guilty_commit.log)
cc @chauhang @penguinwu @chuanqi129 @CaoE