Repository: pytorch
Issue: [inductor][cpu]opacus_cifar10 and functorch_dp_cifar10 AMP performance regression in 2024-11-11 nightly release
Labels: oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>amp static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.783096</td>
      <td>0.010444737999999999</td>
      <td>0.008179232548848</td>
      <td>13.042257</td>
      <td>64</td>
      <td>0.978642</td>
      <td>0.007060253000000001</td>
      <td>0.006909460116426001</td>
      <td>13.063731</td>
      <td>0.8</td>
      <td>0.84</td>
      <td>0.68</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.789183</td>
      <td>0.011084602</td>
      <td>0.008747779460166</td>
      <td>13.726767</td>
      <td>64</td>
      <td>1.230847</td>
      <td>0.00693516</td>
      <td>0.00853612088052</td>
      <td>13.672738</td>
      <td>0.64</td>
      <td>0.98</td>
      <td>0.63</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>5.81558</td>
      <td>0.001893122</td>
      <td>0.011009602440759998</td>
      <td>12.406509</td>
      <td>1</td>
      <td>6.771293</td>
      <td>0.00166472</td>
      <td>0.01127230688296</td>
      <td>12.353326</td>
      <td>0.86</td>
      <td>1.02</td>
      <td>0.88</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>5.713856</td>
      <td>0.001934385</td>
      <td>0.011052797338559999</td>
      <td>13.00355</td>
      <td>1</td>
      <td>6.524588</td>
      <td>0.0016443719999999998</td>
      <td>0.010728849818735998</td>
      <td>12.954403</td>
      <td>0.88</td>
      <td>0.97</td>
      <td>0.85</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

<p>AMP dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.728946</td>
      <td>0.011348397</td>
      <td>0.008272368599562</td>
      <td>15.815295</td>
      <td>64</td>
      <td>1.025937</td>
      <td>0.008005277</td>
      <td>0.008212909869549001</td>
      <td>15.731289</td>
      <td>0.71</td>
      <td>0.99</td>
      <td>0.71</td>
      <td>0.99</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.785165</td>
      <td>0.011229663</td>
      <td>0.008817138349395001</td>
      <td>16.15449</td>
      <td>64</td>
      <td>1.053135</td>
      <td>0.008097649</td>
      <td>0.008527917579615</td>
      <td>16.093056</td>
      <td>0.75</td>
      <td>0.97</td>
      <td>0.72</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>5.796491</td>
      <td>0.0018960539999999999</td>
      <td>0.010990459946513998</td>
      <td>12.354943</td>
      <td>1</td>
      <td>6.658744</td>
      <td>0.00161524</td>
      <td>0.01075546965856</td>
      <td>12.383582</td>
      <td>0.87</td>
      <td>0.98</td>
      <td>0.85</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>5.785104</td>
      <td>0.001917703</td>
      <td>0.011094111296112</td>
      <td>13.000144</td>
      <td>1</td>
      <td>6.664832</td>
      <td>0.001649579</td>
      <td>0.010994166905728</td>
      <td>12.966551</td>
      <td>0.87</td>
      <td>0.99</td>
      <td>0.86</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

<p>AMP dynamic shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.730634</td>
      <td>0.022485953</td>
      <td>0.016429001784202</td>
      <td>35.01237</td>
      <td>64</td>
      <td>0.974152</td>
      <td>0.017241956</td>
      <td>0.016796285921312</td>
      <td>37.339013</td>
      <td>0.75</td>
      <td>1.02</td>
      <td>0.77</td>
      <td>1.07</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.779876</td>
      <td>0.022321979999999998</td>
      <td>0.01740837647448</td>
      <td>33.038597</td>
      <td>64</td>
      <td>1.009547</td>
      <td>0.016893505</td>
      <td>0.017054787292235</td>
      <td>28.392059</td>
      <td>0.77</td>
      <td>0.98</td>
      <td>0.76</td>
      <td>0.86</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>5.055158</td>
      <td>0.0031064919999999998</td>
      <td>0.015703807885736</td>
      <td>58.557283</td>
      <td>1</td>
      <td>5.870757</td>
      <td>0.0026807479999999997</td>
      <td>0.015738020086236</td>
      <td>63.621082</td>
      <td>0.86</td>
      <td>1.0</td>
      <td>0.86</td>
      <td>1.09</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>single</td>
      <td>1</td>
      <td>5.138824</td>
      <td>0.003107214</td>
      <td>0.015967425876336</td>
      <td>24.318755</td>
      <td>1</td>
      <td>6.165533</td>
      <td>0.00269438</td>
      <td>0.01661228880454</td>
      <td>27.084622</td>
      <td>0.83</td>
      <td>1.04</td>
      <td>0.87</td>
      <td>1.11</td>
    </tr>

</table>

<p>AMP static shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>functorch_dp_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.866286</td>
      <td>0.02582328</td>
      <td>0.02237034593808</td>
      <td>44.298414</td>
      <td>64</td>
      <td>1.178419</td>
      <td>0.01879288</td>
      <td>0.022145886856720004</td>
      <td>44.466475</td>
      <td>0.74</td>
      <td>0.99</td>
      <td>0.73</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>opacus_cifar10</td>
      <td>multiple</td>
      <td>64</td>
      <td>0.947846</td>
      <td>0.02319824</td>
      <td>0.021988358991039996</td>
      <td>42.643621</td>
      <td>64</td>
      <td>1.194638</td>
      <td>0.018860679999999998</td>
      <td>0.02253168503384</td>
      <td>42.469943</td>
      <td>0.79</td>
      <td>1.02</td>
      <td>0.81</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

the bad commit a766d84a3c1fe78f246c8e4da2f85b249824151b
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance torchbench opacus_cifar10 amp
Testing with inductor.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  opacus_cifar10
running benchmark: 100%|█████████████████████████████████████████████████████| 50/50 [00:01<00:00, 49.87it/s]
0.568x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,opacus_cifar10,64,0.568399,12.456063,8.959801,0.893319,65.195213,72.980890,71,1,0,0,0,0,1
```
the last good commit 1e9390a30ac29ee3a4a75c184059c5c4cb3d5f0b
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance torchbench opacus_cifar10 amp
Testing with inductor.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  opacus_cifar10
running benchmark: 100%|█████████████████████████████████████████████████████| 50/50 [00:00<00:00, 57.89it/s]
0.745x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,opacus_cifar10,64,0.744655,9.613339,8.972916,0.893434,65.273856,73.059533,71,1,0,0,0,0,1
```


### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>766a5e3a</td>
      <td>main</td>
      <td>e522b45c</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>5ef33e40b3c3fd2608552d3301c7255826c0e7f6</td>
      <td>main</td>
      <td>f121eab0182f7da58b39ffb84744bdc7109817e3</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.5.0a0+fa44bda</td>
      <td>main</td>
      <td>2.5.0a0+fa44bda</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob//main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance torchbench opacus_cifar10 amp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/a766d84a3c1fe78f246c8e4da2f85b249824151b
[torchbench-opacus_cifar10-inference-amp-static-cpp-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/17746417/torchbench-opacus_cifar10-inference-amp-static-cpp-multiple-performance-drop_guilty_commit.log)
cc @chauhang @penguinwu @chuanqi129