Repository: pytorch
Issue: [inductor][cpu]AOT inductor performance regression and accuracy failure in 2024-09-04 nightly release  
Labels: oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>AMP static shape default wrapper performance regression</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>hf_T5_base</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.238967</td>
      <td>0.838228385</td>
      <td>1.0385373074782949</td>
      <td>117.408291</td>
      <td>1</td>
      <td>1.895532</td>
      <td>0.543833995</td>
      <td>1.0308547402103398</td>
      <td>116.595872</td>
      <td>0.65</td>
      <td>0.99</td>
      <td>0.65</td>
      <td>0.99</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>hf_T5_large</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.242756</td>
      <td>0.318922536</td>
      <td>0.39634289514921595</td>
      <td>335.304479</td>
      <td>1</td>
      <td>1.432529</td>
      <td>0.27843835100000003</td>
      <td>0.39887101251967905</td>
      <td>326.659959</td>
      <td>0.87</td>
      <td>1.01</td>
      <td>0.87</td>
      <td>0.97</td>
    </tr>
    <tr>
      <td>huggingface</td>
      <td>MT5ForConditionalGeneration</td>
      <td>multiple</td>
      <td>16</td>
      <td>1.46969</td>
      <td>0.155681585</td>
      <td>0.22880366865865</td>
      <td>101.992322</td>
      <td>16</td>
      <td>1.821134</td>
      <td>0.121095646</td>
      <td>0.220531398182564</td>
      <td>102.407316</td>
      <td>0.81</td>
      <td>0.96</td>
      <td>0.78</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>hf_T5</td>
      <td>single</td>
      <td>1</td>
      <td>0.650519</td>
      <td>0.5984183120000001</td>
      <td>0.389282481903928</td>
      <td>62.38676</td>
      <td>1</td>
      <td>0.880377</td>
      <td>0.441144343</td>
      <td>0.388373333257311</td>
      <td>62.027646</td>
      <td>0.74</td>
      <td>1.0</td>
      <td>0.74</td>
      <td>0.99</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>hf_T5_base</td>
      <td>single</td>
      <td>1</td>
      <td>0.589821</td>
      <td>23.711127984999997</td>
      <td>13.985321219240685</td>
      <td>63.229392</td>
      <td>1</td>
      <td>0.877813</td>
      <td>15.857589405</td>
      <td>13.919998128371265</td>
      <td>65.270571</td>
      <td>0.67</td>
      <td>1.0</td>
      <td>0.67</td>
      <td>1.03</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>hf_T5_large</td>
      <td>single</td>
      <td>1</td>
      <td>0.747827</td>
      <td>5.337602142</td>
      <td>3.991602997045434</td>
      <td>312.96011</td>
      <td>1</td>
      <td>0.98226</td>
      <td>4.059179598</td>
      <td>3.98716975193148</td>
      <td>314.149114</td>
      <td>0.76</td>
      <td>1.0</td>
      <td>0.76</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>huggingface</td>
      <td>T5ForConditionalGeneration</td>
      <td>single</td>
      <td>1</td>
      <td>0.598922</td>
      <td>2.087471869</td>
      <td>1.2502328267252178</td>
      <td>59.535414</td>
      <td>1</td>
      <td>0.869239</td>
      <td>1.434526977</td>
      <td>1.2469467949605029</td>
      <td>59.678655</td>
      <td>0.69</td>
      <td>1.0</td>
      <td>0.69</td>
      <td>1.0</td>
    </tr>
    <tr>
      <td>huggingface</td>
      <td>T5Small</td>
      <td>single</td>
      <td>1</td>
      <td>0.597191</td>
      <td>2.0881833000000003</td>
      <td>1.2470442731103002</td>
      <td>59.967489</td>
      <td>1</td>
      <td>0.873677</td>
      <td>1.433874097</td>
      <td>1.252742819444669</td>
      <td>59.733762</td>
      <td>0.68</td>
      <td>1.0</td>
      <td>0.69</td>
      <td>1.0</td>
    </tr>
  </tbody>

</table>

<p>AMP static default wrapper accuracy failure</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>accuracy</th>
      <th>perf</th>
      <th>reason(reference only)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>timm_models</td>
      <td>levit_128</td>
      <td>multiple</td>
      <td>X</td>
      <td>√</td>
      <td>levit_128, NotImplementedError: Cannot copy out of meta tensor; no data!</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>yolov3</td>
      <td>multiple</td>
      <td>X</td>
      <td>√</td>
      <td>yolov3, NotImplementedError: Cannot copy out of meta tensor; no data!</td>
    </tr>
    <tr>
      <td>timm_models</td>
      <td>levit_128</td>
      <td>single</td>
      <td>X</td>
      <td>√</td>
      <td>levit_128, NotImplementedError: Cannot copy out of meta tensor; no data!</td>
    </tr>
    <tr>
      <td>torchbench</td>
      <td>yolov3</td>
      <td>single</td>
      <td>X</td>
      <td>√</td>
      <td>yolov3, NotImplementedError: Cannot copy out of meta tensor; no data!</td>
    </tr>
  </tbody>

</table>

### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>e000cf0ad9</td>
      <td>main</td>
      <td>2553278bae</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.5.0a0+97ed7b3</td>
      <td>main</td>
      <td>2.4.0a0+b3f6f51</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob//weizhuoz/enable_max_autotune_for_guilty/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance/accuracy **suite** **name** amp first static default 0 aot_inductor
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/387d3fc2968946cbce4b8fbd9315d9bec8b02c9e
[torchbench-hf_T5_base-inference-amp-static-default-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/16940490/torchbench-hf_T5_base-inference-amp-static-default-multiple-performance-drop_guilty_commit.log)
[timm_models-levit_128-inference-amp-static-default-multiple-accuracy-crash_guilty_commit.log](https://github.com/user-attachments/files/16940496/timm_models-levit_128-inference-amp-static-default-multiple-accuracy-crash_guilty_commit.log)
cc @ezyang @chauhang @penguinwu @WeizhuoZhang-intel @chuanqi129 @chunyuan-w