Repository: pytorch
Issue: [inductor][cpu] AMP CPP wrapper performance regression in 2024-11-25 nightly release
Labels: oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>amp static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>huggingface</td>
      <td>MegatronBertForQuestionAnswering</td>
      <td>multiple</td>
      <td>8</td>
      <td>2.056239</td>
      <td>0.167259388</td>
      <td>0.34392527672173206</td>
      <td>26.978458</td>
      <td>8</td>
      <td>2.679464</td>
      <td>0.129058642</td>
      <td>0.34580798512788796</td>
      <td>25.750905</td>
      <td>0.77</td>
      <td>1.01</td>
      <td>0.77</td>
      <td>0.95</td>
    </tr>
  </tbody>

</table>

<p>amp dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>hf_Bert_large</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.033177</td>
      <td>0.084021655</td>
      <td>0.086809241447935</td>
      <td>26.319139</td>
      <td>1</td>
      <td>2.77511</td>
      <td>0.033001139</td>
      <td>0.09158179085029</td>
      <td>25.424551</td>
      <td>0.37</td>
      <td>1.05</td>
      <td>0.39</td>
      <td>0.97</td>
    </tr>
    <tr>
      <td>huggingface</td>
      <td>MegatronBertForQuestionAnswering</td>
      <td>multiple</td>
      <td>8</td>
      <td>2.103844</td>
      <td>0.164229288</td>
      <td>0.345512802183072</td>
      <td>38.84587</td>
      <td>8</td>
      <td>2.653344</td>
      <td>0.12995433399999998</td>
      <td>0.34481355239289596</td>
      <td>37.212765</td>
      <td>0.79</td>
      <td>1.0</td>
      <td>0.79</td>
      <td>0.96</td>
    </tr>
  </tbody>

</table>

the last good commit: 2fc692b3dd42bf92c4f92dcec862bae7ae1c7995
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance huggingface MegatronBertForQuestionAnswering amp first static cpp
Testing with cpp wrapper.
Testing with inductor.
multi-threads testing....
/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
loading model: 0it [00:03, ?it/s]
cpu  eval  MegatronBertForQuestionAnswering
running benchmark: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:23<00:00,  2.13it/s]
2.632x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,MegatronBertForQuestionAnswering,8,2.632258,128.201069,28.769934,0.936615,1302.990029,1391.169126,726,1,0,0,0,0,0
```
the bad commit: 819b0ebd944c5fca4188541adcbd53f87a2ce534
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance huggingface MegatronBertForQuestionAnswering amp first static cpp
Testing with cpp wrapper.
Testing with inductor.
multi-threads testing....
/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
loading model: 0it [00:03, ?it/s]
cpu  eval  MegatronBertForQuestionAnswering
running benchmark: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:25<00:00,  1.98it/s]
1.996x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,MegatronBertForQuestionAnswering,8,1.995862,168.099244,29.314211,0.726404,1303.114138,1793.925120,726,1,0,0,0,0,0
```


### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>766a5e3a</td>
      <td>main</td>
      <td>766a5e3a</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>c513f01516673898d551818c8ca6085cf07e4006</td>
      <td>main</td>
      <td>2fc692b3dd42bf92c4f92dcec862bae7ae1c7995</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.5.0a0+332760d</td>
      <td>main</td>
      <td>2.5.0a0+332760d</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob//main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance huggingface MegatronBertForQuestionAnswering amp first static cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/819b0ebd944c5fca4188541adcbd53f87a2ce534
[huggingface-MegatronBertForQuestionAnswering-inference-amp-static-cpp-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/17930843/huggingface-MegatronBertForQuestionAnswering-inference-amp-static-cpp-multiple-performance-drop_guilty_commit.log)
cc @chauhang @penguinwu @chuanqi129