Repository: pytorch
Issue: [Inductor] [CPU] Huggingface model AllenaiLongformerBase & MT5ForConditionalGeneration &T5ForConditionalGeneration & T5Small & hf_Longformer  performance regression > 10% on ww07.4
Labels: triaged,module: inductor,oncall: cpu inductor

## 🐛 Describe the bug

### Single Thread

There is a performance regression on Huggingface model **AllenaiLongformerBase** and **hf_Longformer** on ww07.4 [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531#issuecomment-1434019577)  compare with ww07.2 as bellow:

#### AllenaiLongformerBase

|  ww07.4   |   |     |   |  ww07.2   |   |     |   | Comp  |     |   |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  |
| batch_size  | speedup | inductor  | eager | batch_size  | speedup | inductor  | eager | speedup ratio | eager ratio  | inductor ratio |
|1	|0.9901	|2.3819678	|2.358386319	|1	|1.1392	|2.0668015	|2.354500269	|0.87	|1	|0.87

Please find sw info from [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531) 

Graph dump by cosim:

WW07.4:
[graph.txt](https://github.com/pytorch/pytorch/files/10765350/graph.txt)

WW07.2:
[graph.txt](https://github.com/pytorch/pytorch/files/10765352/graph.txt)

Repro:

```
python -m torch.backends.xeon.run_cpu --core_list 0 --ncores_per_instance 1 benchmarks/dynamo/huggingface.py --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only AllenaiLongformerBase  --cold_start_latency --batch_size 1 --threads 1
```
#### hf_Longformer

|  ww07.4   |   |     |   |  ww07.2   |   |     |   | Comp  |     |   |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  |
| batch_size  | speedup | inductor  | eager | batch_size  | speedup | inductor  | eager | speedup ratio | eager ratio  | inductor ratio |
|1	|1.0426	|1.0742891	|1.120053816	|1	|1.1938	|0.943502	 |1.126352688	|0.87	|1.01	|0.88

Please find sw info from [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531) 

Graph dump by cosim:

WW07.4:
[graph.txt](https://github.com/pytorch/pytorch/files/10765423/graph.txt)


WW07.2:
[graph.txt](https://github.com/pytorch/pytorch/files/10765435/graph.txt)

Repro:

```
python -m torch.backends.xeon.run_cpu --core_list 0 --ncores_per_instance 1 benchmarks/dynamo/torchbench.py --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only hf_Longformer  --cold_start_latency --batch_size 1 --threads 1
```

### Multi Threads

There is a performance regression on Huggingface models **MT5ForConditionalGeneration &T5ForConditionalGeneration & T5Small** on ww07.4 [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531#issuecomment-1434019397)  compare with ww07.2 as bellow:

| |  ww07.4   |   |     |   |  ww07.2   |   |     |   | Comp  |     |   |
| ---- |  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  |
| name | batch_size  | speedup | inductor  | eager | batch_size  | speedup | inductor  | eager | speedup ratio | eager ratio  | inductor ratio |
|MT5ForConditionalGeneration	|16	|1.0417	|0.2628685	|0.273830116	|16	|1.1846	|0.2305143	|0.27306724	|0.88	|1	|0.88
|T5ForConditionalGeneration	|4	|0.7722	|0.5160617	|0.398502845	|4	|1.0301	|0.3849303	|0.396516702	|0.75	|1	|0.75
|T5Small	|4	|0.7736	|0.5160818	|0.39924088	|4	|1.0333	|0.3849365	|0.397754885	|0.75	|1	|0.75

Please find sw info from [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531) 

Graph dump by cosim:

WW07.4:
[MT5ForConditionalGeneration graph.txt](https://github.com/pytorch/pytorch/files/10765405/graph.txt)
[T5ForConditionalGeneration graph.txt](https://github.com/pytorch/pytorch/files/10765393/graph.txt)
[T5Small graph.txt](https://github.com/pytorch/pytorch/files/10765363/graph.txt)

WW07.2:
[MT5ForConditionalGeneration graph.txt](https://github.com/pytorch/pytorch/files/10765406/graph.txt)
[T5ForConditionalGeneration graph.txt](https://github.com/pytorch/pytorch/files/10765396/graph.txt)
[T5Small graph.txt](https://github.com/pytorch/pytorch/files/10765367/graph.txt)

Repro:

```
python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/huggingface.py --performance --float32 -dcpu  -n50 --inductor  --no-skip --dashboard --only MODEL  --cold_start_latency
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @soumith @yanboliang @anijain2305 @Xia-Weiwen @desertfire