Repository: pytorch
Issue: [inductor][cpu]MT5ForConditionalGeneration, T5ForConditionalGeneration and T5Small performance regression in 2024-07-15 nightly release
Labels: triaged,oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>AMP dynamic shape CPP wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>huggingface</td>
      <td>MT5ForConditionalGeneration</td>
      <td>multiple</td>
      <td>16</td>
      <td>2.110904</td>
      <td>0.066520629</td>
      <td>0.140418661838616</td>
      <td>59.831839</td>
      <td>16</td>
      <td>2.338585</td>
      <td>0.059047208</td>
      <td>0.13808691492068</td>
      <td>58.435523</td>
      <td>0.9</td>
      <td>0.98</td>
      <td>0.89</td>
      <td>0.98</td>
    </tr>
    <tr>
      <td>huggingface</td>
      <td>T5ForConditionalGeneration</td>
      <td>multiple</td>
      <td>4</td>
      <td>2.373645</td>
      <td>0.125174213</td>
      <td>0.29711914481638496</td>
      <td>42.516591</td>
      <td>4</td>
      <td>2.74185</td>
      <td>0.104517613</td>
      <td>0.28657161720405</td>
      <td>41.061006</td>
      <td>0.87</td>
      <td>0.96</td>
      <td>0.83</td>
      <td>0.97</td>
    </tr>
    <tr>
      <td>huggingface</td>
      <td>T5Small</td>
      <td>multiple</td>
      <td>4</td>
      <td>2.351207</td>
      <td>0.12452803900000001</td>
      <td>0.29279119699307304</td>
      <td>42.802429</td>
      <td>4</td>
      <td>2.739233</td>
      <td>0.104945335</td>
      <td>0.287469724828055</td>
      <td>41.186878</td>
      <td>0.86</td>
      <td>0.98</td>
      <td>0.84</td>
      <td>0.96</td>
    </tr>
  </tbody>

</table>

Bad Commit:
b7d287fbec0a05a3d4c9524006e6bfd1de6a71a0
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance huggingface T5ForConditionalGeneration amp first dynamic cpp
Testing with dynamic shapes.
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
/opt/conda/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
loading model: 0it [00:01, ?it/s]
cpu  eval  T5ForConditionalGeneration
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:22<00:00,  2.25it/s]
1.946x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,T5ForConditionalGeneration,4,1.945825,149.929709,44.395475,0.972070,825.967002,849.698816,813,1,0,0,0,0,0

/workspace/pytorch# bash inductor_single_run.sh multiple inference performance huggingface T5Small amp first dynamic cpp
Testing with dynamic shapes.
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
/opt/conda/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
loading model: 0it [00:01, ?it/s]
cpu  eval  T5Small
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:22<00:00,  2.24it/s]
1.979x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,T5Small,4,1.979264,149.242789,43.755890,0.972172,826.225459,849.875763,813,1,0,0,0,0,0
```

The last Good Commit:
ae0edadea039406902d9656072908eaf3182a0f6
```
/workspace/pytorch# bash inductor_single_run.sh multiple inference performance huggingface T5ForConditionalGeneration amp first dynamic cpp
Testing with dynamic shapes.
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
/opt/conda/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
loading model: 0it [00:01, ?it/s]
cpu  eval  T5ForConditionalGeneration
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:20<00:00,  2.44it/s]
2.506x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,T5ForConditionalGeneration,4,2.506050,116.399641,43.265102,0.971706,825.910886,849.959322,813,1,0,0,0,0,0

/workspace/pytorch# bash inductor_single_run.sh multiple inference performance huggingface T5Small amp first dynamic cpp
Testing with dynamic shapes.
Testing with cpp wrapper.
Testing with freezing on.
multi-threads testing....
/opt/conda/lib/python3.8/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
loading model: 0it [00:01, ?it/s]
cpu  eval  T5Small
running benchmark: 100%|█████████████████████████████████████████████████████████████████| 50/50 [00:19<00:00,  2.52it/s]
2.735x
WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
dev,name,batch_size,speedup,abs_latency,compilation_latency,compression_ratio,eager_peak_mem,dynamo_peak_mem,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks,autograd_captures,autograd_compiles,cudagraph_skips
cpu,T5Small,4,2.735039,105.347396,42.949334,0.968460,825.832243,852.726989,813,1,0,0,0,0,0
```



### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>9ab8d47f9d27f6f4102e7786fcd604861d8e480c</td>
      <td>main</td>
      <td>7128504424ca54311efdf22f2c8425291586860e</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.4.0a0+69b2a0a</td>
      <td>main</td>
      <td>2.4.0a0+b829e93</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance huggingface T5Small amp first dynamic cpp
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/b7d287fbec0a05a3d4c9524006e6bfd1de6a71a0
[huggingface-T5Small-inference-amp-dynamic-cpp-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/16270375/huggingface-T5Small-inference-amp-dynamic-cpp-multiple-performance-drop_guilty_commit.log)
cc @ezyang @anijain2305 @chauhang @penguinwu @WeizhuoZhang-intel @chuanqi129