Repository: pytorch
Issue: [inductor][cpu]pyhpc_isoneutral_mixing singel thread AMP cpp wrapper performance regression in  2024-09-22 nightly release
Labels: oncall: pt2,module: inductor,oncall: cpu inductor

### 🐛 Describe the bug

<p>AMP static shape CPP wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>pyhpc_isoneutral_mixing</td>
      <td>single</td>
      <td>1</td>
      <td>35.939996</td>
      <td>4.5546999999999996e-05</td>
      <td>0.001636958997812</td>
      <td>16.185845</td>
      <td>1</td>
      <td>41.469536</td>
      <td>3.9793000000000003e-05</td>
      <td>0.001650197246048</td>
      <td>16.201631</td>
      <td>0.87</td>
      <td>1.01</td>
      <td>0.87</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

<p>AMP dynamic shape CPP wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>pyhpc_isoneutral_mixing</td>
      <td>single</td>
      <td>1</td>
      <td>34.470612</td>
      <td>4.7884000000000004e-05</td>
      <td>0.0016505907850080002</td>
      <td>16.190656</td>
      <td>1</td>
      <td>41.298161</td>
      <td>4.05e-05</td>
      <td>0.0016725755205</td>
      <td>16.127852</td>
      <td>0.83</td>
      <td>1.01</td>
      <td>0.85</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

The bad commit: cdef760560049ebda5fb7e30b1703f345fe05cfa
```
/workspace/pytorch# bash inductor_single_run.sh single inference performance torchbench pyhpc_isoneutral_mixing amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  pyhpc_isoneutral_mixing
running benchmark: 100%|██████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 520.46it/s]
33.564x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,pyhpc_isoneutral_mixing,1,33.564253,0.048496,8.654499,0.810945,39.342080,48.513843,746,1,0,0,0,0,0
```
The last good commit: 4842f0fac6783c4a0354bb674d33dfa78a392e47
```
/workspace/pytorch# bash inductor_single_run.sh single inference performance torchbench pyhpc_isoneutral_mixing amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  pyhpc_isoneutral_mixing
running benchmark: 100%|██████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 517.47it/s]
39.422x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,pyhpc_isoneutral_mixing,1,39.421643,0.041728,8.581601,0.812124,39.400243,48.515072,746,1,0,0,0,0,0
```
</table>


### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>9731ccb9e0c9f09dcd3435a47fc40fe6fb960504</td>
      <td>main</td>
      <td>c64ae601ba9eb3ad2cd3402a14f6ac83c0ab7eba</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.5.0a0+ba696ea</td>
      <td>main</td>
      <td>2.5.0a0+97ed7b3</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob//main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh single inference performance torchbench pyhpc_isoneutral_mixing amp first static cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/cdef760560049ebda5fb7e30b1703f345fe05cfa
cc @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @WeizhuoZhang-intel @chuanqi129
[torchbench-pyhpc_isoneutral_mixing-inference-amp-static-cpp-single-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/17128779/torchbench-pyhpc_isoneutral_mixing-inference-amp-static-cpp-single-performance-drop_guilty_commit.log)