Repository: pytorch
Issue: [CPU] jx_nest_base AMP both inductor and eager performance regression in 2024-08-10 nightly release
Labels: triaged,oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>amp static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>timm_models</td>
      <td>jx_nest_base</td>
      <td>multiple</td>
      <td>32</td>
      <td>1.634363</td>
      <td>0.14318641</td>
      <td>0.23401857060682998</td>
      <td>45.792037</td>
      <td>32.0</td>
      <td>1.476906</td>
      <td>0.09909578000000001</td>
      <td>0.14635515205668</td>
      <td>46.69523</td>
      <td>1.11</td>
      <td>0.63</td>
      <td>0.69</td>
      <td>1.02</td>
    </tr>
  </tbody>

</table>

<p>amp static shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>timm_models</td>
      <td>jx_nest_base</td>
      <td>multiple</td>
      <td>32</td>
      <td>1.701064</td>
      <td>0.15762912199999998</td>
      <td>0.268137224785808</td>
      <td>51.498577</td>
      <td>32</td>
      <td>1.46995</td>
      <td>0.11794990799999999</td>
      <td>0.1733804672646</td>
      <td>50.041447</td>
      <td>1.16</td>
      <td>0.65</td>
      <td>0.75</td>
      <td>0.97</td>
    </tr>
  </tbody>

</table>

the bad commit:
c7cfa5172139737bf75afbd4a7920b1a02b1dcb2
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance timm_models jx_nest_base amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
loading model: 0it [00:01, ?it/s]
cpu  eval  jx_nest_base
running benchmark: 100%|████████████████████████████████████████████████████| 50/50 [00:18<00:00,  2.69it/s]
1.621x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,jx_nest_base,32,1.620818,141.661849,51.562021,0.983694,896.307200,911.164621,553,1,0,0,0,0,0
```

</table>

the last good commit 
01cdcbf7c83cd84a3685766f7f8cd26ad447feae
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance timm_models jx_nest_base amp first static cpp
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
loading model: 0it [00:01, ?it/s]
cpu  eval  jx_nest_base
running benchmark: 100%|████████████████████████████████████████████████████| 50/50 [00:12<00:00,  4.16it/s]
1.437x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,jx_nest_base,32,1.436553,98.053498,47.794398,0.979158,693.798912,708.566630,553,1,0,0,0,0,0
```

### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>a7912bf9dc39b934baf5e04b436cc2134776c10d</td>
      <td>main</td>
      <td>6ec4af6865dd884f984c9dbcb273ae26e3825481</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.4.0a0+b3f6f51</td>
      <td>main</td>
      <td>2.4.0a0+b3f6f51</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
      <td>main</td>
      <td>0.7.0a0+11bb5b8</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance timm_models jx_nest_base amp first static cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/c7cfa5172139737bf75afbd4a7920b1a02b1dcb2
[timm_models-jx_nest_base-inference-amp-static-cpp-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/16615577/timm_models-jx_nest_base-inference-amp-static-cpp-multiple-performance-drop_guilty_commit.log)
cc @ezyang @chauhang @penguinwu @WeizhuoZhang-intel @chuanqi129