Repository: pytorch
Issue: Python 3.10 + intel_openmp shows performance regression when testing benchmarks/dynamo
Labels: triaged,oncall: pt2

### 🐛 Describe the bug

We are planning upgrading our python environment from 3.8 to 3.10, because pytorch has deprecated python 3.8 recently.
But we found that there are performance gaps between python 3.8 and 3.10 when testing benchmarks/dynamo suites.
After debugging and try different configurations, we found that it might related to `libiomp5.so` and environment variable `KMP_AFFINITY`.

**Performance**
|     Python    	|     Dev    	|     name            	|     batch_size    	|     speedup     	|     abs_latency     	|     compilation_latency    	|
|---------------	|------------	|---------------------	|-------------------	|-----------------	|---------------------	|----------------------------	|
|     3.8       	|     Cpu    	|     BERT_pytorch    	|     2             	|     1.83045     	|     **21.87665**    	|     28.22332               	|
|     3.10      	|     Cpu    	|     BERT_pytorch    	|     2             	|     2.509247    	|     **403.2253**    	|     25.30055               	|

**Observations:**

- When `LD_PRELOAD=libiomp5.so` and `KMP_AFFINITY` are set, it shows performance regression. Unset one of them, performance is back to normal.

```shell
export LD_PRELOAD=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libiomp5.so
export KMP_AFFINITY=granularity=fine,compact,1,0
```

- And we found that above configurations are set, there is only 1 core running on process `pt_main_thread` which might be the root cause of this poor performance on Python 3.10.

![Screenshot 2024-08-09 110118](https://github.com/user-attachments/assets/8c69a399-f1a6-4279-8a38-d5705696c83e)

- It can only be reproduce by executing all the model for each `suite.py`, which means that we cannot reproduce by executing only 1 model by passing `--only BERT_pytorch`
```shell
numactl -C 0-39 -m 0 python dynamo/torchbench.py --performance --amp -dcpu --output=/home/pytorch/pytorch/benchmarks/test.csv --inference -n50 --inductor --timeout 9000 --freezing --no-skip --dashboard --only BERT_pytorch --cold-start-latency
```

**How to reproduce:**
Python 3.8
```shell
conda create -n pt_38 python=3.8
conda activate pt_38
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
# pip list | grep torch
# torch             2.5.0.dev20240729+cpu
# torchaudio        2.4.0.dev20240729+cpu
# torchvision       0.20.0.dev20240729+cpu
git clone --depth 1 https://github.com/pytorch/pytorch.git
git clone --depth 1 https://github.com/pytorch/benchmark.git
cd benchmark
pip install --no-deps -r requirements.txt
python install.py BERT_pytorch
cd ../pytorch/benchmarks/
pip install intel-openmp pandas scipy
export LD_PRELOAD=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libiomp5.so
export KMP_AFFINITY=granularity=fine,compact,1,0
# md5sum /home/pytorch/miniforge3/envs/pt_38/lib/libiomp5.so
# 52ca1660535eb2464207d9f2e2ee0453  /home/pytorch/miniforge3/envs/pt_38/lib/libiomp5.so
# pip list | grep intel-openmp
# intel-openmp       2024.2.1
numactl -C 0-39 -m 0 python dynamo/torchbench.py --performance --amp -dcpu --output=/home/pytorch/pytorch/benchmarks/test.csv --inference -n50 --inductor --timeout 9000 --freezing --no-skip --dashboard -x torchrec_dlrm --cold-start-latency
```

Python 3.10
```shell
conda create -n pt_310 python=3.10
conda activate pt_310
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
# pip list | grep torch
# torch             2.5.0.dev20240808+cpu
# torchaudio        2.4.0.dev20240808+cpu
# torchvision       0.20.0.dev20240808+cpu
git clone --depth 1 https://github.com/pytorch/pytorch.git
git clone --depth 1 https://github.com/pytorch/benchmark.git
cd benchmark
pip install --no-deps -r requirements.txt
python install.py BERT_pytorch
cd ../pytorch/benchmarks/
pip install intel-openmp pandas scipy numpy==1.26
export LD_PRELOAD=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libiomp5.so
export KMP_AFFINITY=granularity=fine,compact,1,0
# md5sum /home/pytorch/miniforge3/envs/pt_310/lib/libiomp5.so
# 52ca1660535eb2464207d9f2e2ee0453  /home/pytorch/miniforge3/envs/pt_310/lib/libiomp5.so
# pip list | grep intel-openmp
# intel-openmp       2024.2.1
numactl -C 0-39 -m 0 python dynamo/torchbench.py --performance --amp -dcpu --output=/home/pytorch/pytorch/benchmarks/test.csv --inference -n50 --inductor --timeout 9000 --freezing --no-skip --dashboard -x torchrec_dlrm --cold-start-latency
```

### Versions

**Versions**
```shell
Python 3.8
# pip list | grep torch
# torch             2.5.0.dev20240729+cpu
# torchaudio        2.4.0.dev20240729+cpu
# torchvision       0.20.0.dev20240729+cpu

python3.10
# pip list | grep torch
# torch             2.5.0.dev20240808+cpu
# torchaudio        2.4.0.dev20240808+cpu
# torchvision       0.20.0.dev20240808+cpu
```

cc @ezyang @chauhang @penguinwu