Repository: pytorch
Issue: [inductor][cpu] pyhpc_turbulent_kinetic_energy and pyhpc_isoneutral_mixing amp performance regression in 2024-08-04 nightly release
Labels: triaged,oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>AMP static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>pyhpc_isoneutral_mixing</td>
      <td>multiple</td>
      <td>1048576.0</td>
      <td>0.807968</td>
      <td>0.03564787</td>
      <td>0.028802338228159998</td>
      <td>47.734733</td>
      <td>1048576</td>
      <td>0.932513</td>
      <td>0.030912795</td>
      <td>0.028826583203835</td>
      <td>30.996681</td>
      <td>0.87</td>
      <td>1.0</td>
      <td>0.87</td>
      <td>0.65</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>pyhpc_turbulent_kinetic_energy</td>
      <td>multiple</td>
      <td>1048576.0</td>
      <td>0.744818</td>
      <td>0.058364756999999996</td>
      <td>0.043471121579225995</td>
      <td>43.053543</td>
      <td>1048576</td>
      <td>1.516383</td>
      <td>0.028633226</td>
      <td>0.043418937141558006</td>
      <td>27.386363</td>
      <td>0.49</td>
      <td>1.0</td>
      <td>0.49</td>
      <td>0.64</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>pyhpc_turbulent_kinetic_energy</td>
      <td>single</td>
      <td>1048576</td>
      <td>0.880334</td>
      <td>0.338152554</td>
      <td>0.297687190473036</td>
      <td>44.087438</td>
      <td>1048576</td>
      <td>1.633465</td>
      <td>0.181848323</td>
      <td>0.297042870929195</td>
      <td>26.86572</td>
      <td>0.54</td>
      <td>1.0</td>
      <td>0.54</td>
      <td>0.61</td>
    </tr>
  </tbody>

</table>

The bad commit:
6ec4af6865dd884f984c9dbcb273ae26e3825481
```
/workspace/pytorch# bash inductor_single_run.sh  multiple inference performance torchbench pyhpc_turbulent_kinetic_energy amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  pyhpc_turbulent_kinetic_energy
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:04<00:00, 10.16it/s]
0.697x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,pyhpc_turbulent_kinetic_energy,1048576,0.697260,57.291189,46.769229,0.752425,106.780672,141.915341,996,1,0,0,0,0,0

/workspace/pytorch# bash inductor_single_run.sh  multiple inference performance torchbench pyhpc_isoneutral_mixing amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  pyhpc_isoneutral_mixing
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:03<00:00, 16.16it/s]
0.715x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,pyhpc_isoneutral_mixing,1048576,0.715237,35.433943,48.260618,0.557938,115.720602,207.407514,746,1,0,0,0,0,0
```

The last good commit:
d9841057486db5db04a0e0aace2fa3c19b109389
```
/workspace/pytorch# bash inductor_single_run.sh  multiple inference performance torchbench pyhpc_turbulent_kinetic_energy amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  pyhpc_turbulent_kinetic_energy
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:03<00:00, 14.56it/s]
1.450x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,pyhpc_turbulent_kinetic_energy,1048576,1.449987,27.603463,27.867218,0.753777,106.606592,141.429965,996,1,0,0,0,0,0

/workspace/pytorch# bash inductor_single_run.sh  multiple inference performance torchbench pyhpc_isoneutral_mixing amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  pyhpc_isoneutral_mixing
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:02<00:00, 17.32it/s]
0.825x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,pyhpc_isoneutral_mixing,1048576,0.825316,31.128649,31.629833,0.560098,115.725107,206.615757,746,1,0,0,0,0,0
```

### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>c7cfa5172139737bf75afbd4a7920b1a02b1dcb2</td>
      <td>main</td>
      <td>1d1d074072ecb0aa6ca95e3f43221d2275e16d74</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.4.0a0+b3f6f51</td>
      <td>main</td>
      <td>2.4.0a0+69b2a0a</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh  multiple inference performance torchbench pyhpc_turbulent_kinetic_energy amp first static cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/6ec4af6865dd884f984c9dbcb273ae26e3825481
[torchbench-pyhpc_turbulent_kinetic_energy-inference-amp-static-default-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/16536558/torchbench-pyhpc_turbulent_kinetic_energy-inference-amp-static-default-multiple-performance-drop_guilty_commit.log)
cc @ezyang @chauhang @penguinwu @WeizhuoZhang-intel @chuanqi129