Repository: pytorch
Issue: [inductor][cpu] hf_BigBird float32 multiple threads performance regression in 2024-09-08 nightly release
Labels: oncall: pt2,module: inductor,oncall: cpu inductor

### 🐛 Describe the bug

<p>float32 dynamic shape default wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>hf_BigBird</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.00978</td>
      <td>0.168430035</td>
      <td>0.1700772807423</td>
      <td>51.089231</td>
      <td>1</td>
      <td>1.172052</td>
      <td>0.144673477</td>
      <td>0.16956483806480402</td>
      <td>66.752075</td>
      <td>0.86</td>
      <td>1.0</td>
      <td>0.86</td>
      <td>1.31</td>
    </tr>
  </tbody>

</table>

<p>float32 static shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>hf_BigBird</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.005213</td>
      <td>0.161506732</td>
      <td>0.16234866659391597</td>
      <td>330.19123</td>
      <td>1</td>
      <td>1.182942</td>
      <td>0.13820362600000002</td>
      <td>0.16348687374769202</td>
      <td>928.454535</td>
      <td>0.85</td>
      <td>1.01</td>
      <td>0.86</td>
      <td>2.81</td>
    </tr>
  </tbody>

</table>

<p>float32 dynamic shape cpp wrapper</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>suite</th>
      <th>name</th>
      <th>thread</th>
      <th>batch_size_new</th>
      <th>speed_up_new</th>
      <th>inductor_new</th>
      <th>eager_new</th>
      <th>compilation_latency_new</th>
      <th>batch_size_old</th>
      <th>speed_up_old</th>
      <th>inductor_old</th>
      <th>eager_old</th>
      <th>compilation_latency_old</th>
      <th>Ratio Speedup(New/old)</th>
      <th>Eager Ratio(old/new)</th>
      <th>Inductor Ratio(old/new)</th>
      <th>Compilation_latency_Ratio(old/new)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>hf_BigBird</td>
      <td>multiple</td>
      <td>1</td>
      <td>1.004326</td>
      <td>0.161354734</td>
      <td>0.162052754579284</td>
      <td>335.387007</td>
      <td>1</td>
      <td>1.178599</td>
      <td>0.137564144</td>
      <td>0.162132962554256</td>
      <td>943.997328</td>
      <td>0.85</td>
      <td>1.0</td>
      <td>0.85</td>
      <td>2.81</td>
    </tr>
  </tbody>

</table>

### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>name</th>
      <th>target_branch</th>
      <th>target_commit</th>
      <th>refer_branch</th>
      <th>refer_commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>main</td>
      <td>23512dbe</td>
      <td>main</td>
      <td>23512dbe</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>main</td>
      <td>3bebc09be9845c0779f190489e8d4caa9e2653c8</td>
      <td>main</td>
      <td>c140fa1426603322a5a69ef91300f13489db5970</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
      <td>main</td>
      <td>0.19.0a0+d23a6e1</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
      <td>main</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>main</td>
      <td>2.5.0a0+97ed7b3</td>
      <td>main</td>
      <td>2.5.0a0+97ed7b3</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
      <td>main</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>main</td>
      <td>nightly</td>
      <td>main</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
[inductor_single_run.sh](https://github.com/chuanqi129/inductor-tools/blob/main/scripts/modelbench/inductor_single_run.sh)
bash inductor_single_run.sh multiple inference performance torchbench hf_BigBird float32 first static cpp
[torchbench-hf_BigBird-inference-float32-static-cpp-multiple-performance-drop_guilty_commit.log](https://github.com/user-attachments/files/16962246/torchbench-hf_BigBird-inference-float32-static-cpp-multiple-performance-drop_guilty_commit.log)
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/95e976a63f0979a73ee2decb9092babeea55c13e
cc @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @WeizhuoZhang-intel @chuanqi129