Repository: pytorch
Issue: [inductor][cpu]opacus_cifar10 performance regression in 2024-07-15 nightly release
Labels: triaged,oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>AMP dynamic shape CPP wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>0.901115</td>
      <td>0.010627216</td>
      <td>0.00957634374584</td>
      <td>18.958285</td>
      <td>1</td>
      <td>5.780103</td>
      <td>0.001830805</td>
      <td>0.010582241472915001</td>
      <td>19.029104</td>
      <td>0.16</td>
      <td>1.11</td>
      <td>0.17</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

<p>AMP static shape CPP wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.568411</td>
      <td>0.013150195</td>
      <td>0.007474715490145</td>
      <td>19.473226</td>
      <td>64</td>
      <td>0.658368</td>
      <td>0.011227968999999999</td>
      <td>0.007392135494591999</td>
      <td>19.491732</td>
      <td>0.86</td>
      <td>0.99</td>
      <td>0.85</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>0.909894</td>
      <td>0.010711930000000001</td>
      <td>0.009746720835420001</td>
      <td>18.971047</td>
      <td>1</td>
      <td>5.831907</td>
      <td>0.001825543</td>
      <td>0.010646397000501</td>
      <td>18.969377</td>
      <td>0.16</td>
      <td>1.09</td>
      <td>0.17</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

<p>float32 static shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>0.511783</td>
      <td>0.010803611999999999</td>
      <td>0.005529104960195999</td>
      <td>26.53207</td>
      <td>1</td>
      <td>2.194856</td>
      <td>0.002569617</td>
      <td>0.005639939290152001</td>
      <td>9.800167</td>
      <td>0.23</td>
      <td>1.02</td>
      <td>0.24</td>
      <td>0.37</td>
    </tr>
  </tbody>

</table>

The Bad Commit:
f7d7b94017f49a01f51da6167b7180e28fb835d0
```
/workspace/pytorch# bash inductor_single_run.sh single inference performance torchbench opacus_cifar10 amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  opacus_cifar10
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:01<00:00, 47.81it/s]
0.907x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,opacus_cifar10,1,0.907084,10.716454,19.506218,0.882750,65.721958,74.451354,71,1,0,0,0,0,0
```

The last good commit:
fed8b0055f599463f66a922213f6081756dfb77b
```
/workspace/pytorch# bash inductor_single_run.sh single inference performance torchbench opacus_cifar10 amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  opacus_cifar10
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 76.66it/s]
5.756x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,opacus_cifar10,1,5.755827,1.832739,19.620678,0.888535,65.824358,74.081894,71,1,0,0,0,0,0
```


### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>f9f85bfc0b5b63274fa3fdd22afb0a456abf53f4</td>
      <td>main</td>
      <td>007e75958fd2e822b5266df500cf791c14b92b62</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.4.0a0+69b2a0a</td>
      <td>main</td>
      <td>2.4.0a0+69b2a0a</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>fea73cb</td>
      <td>main</td>
      <td>fea73cb</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh single inference performance torchbench opacus_cifar10 amp first static cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/f7d7b94017f49a01f51da6167b7180e28fb835d0
[torchbench-opacus_cifar10-inference-float32-static-default-single-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/16268349/torchbench-opacus_cifar10-inference-float32-static-default-single-performance-drop_guilty_commit.log)
cc @ezyang @anijain2305 @chauhang @penguinwu @WeizhuoZhang-intel @chuanqi129