Repository: pytorch
Issue: [inductor][cpu]shufflenet_v2_x1_0 QAT performance regression in 2024-04-20 nightly release
Labels: oncall: pt2,oncall: cpu inductor

### 🐛 Describe the bug

<p>shufflenet_v2_x1_0 QAT performance regression</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>model_name</th>
      <th>qat_new</th>
      <th>qat_old</th>
      <th>qat ratio(new/old)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>shufflenet_v2_x1_0-eval_throughput</td>
      <td>5728.255241029094</td>
      <td>6910.654523206949</td>
      <td>0.83</td>
    </tr>
  </tbody>


</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>SW</th>
      <th>Nightly commit</th>
      <th>Main commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Pytorch</td>
      <td>02b1ebb</td>
      <td></td>
    </tr>
    <tr>
      <td>Torchbench</td>
      <td>/</td>
      <td>ee35d764</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>ea437b3</td>
      <td></td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>b0ebddc</td>
      <td></td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>2c4665f</td>
      <td></td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>0790338</td>
      <td></td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>nightly</td>
      <td>/</td>
    </tr>
  </tbody>
</table><p>Reference SW info (nightly)</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>item</th>
      <th>commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>torchbench</td>
      <td>ee35d764</td>
    </tr>
    <tr>
      <td>torch</td>
      <td>2.4.0a0+gitd8b31eb</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>0.19.0a0+2c4665f</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>0.16.0a0+b0ebddc</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>2.2.0a0+ea437b3</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>0.7.1a0+0790338</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>nightly</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
```
git clone -b chuanqiw/inductor_quant https://github.com/pytorch/benchmark.git
cd benchmark
pip install --no-deps -r requirements.txt
pip install --no-cache Jinja2==3.1.2 markupsafe==2.0.1 beartype==0.15.0 && pip install mpmath==1.3.0
python install.py --continue_on_fail
export LD_PRELOAD=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libiomp5.so:${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libjemalloc.so
export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:-1,muzzy_decay_ms:-1"
#QAT
TORCHINDUCTOR_FREEZING=1 python run_benchmark.py cpu -m shufflenet_v2_x1_0 --torchdynamo inductor --quantize --is_qat --launcher --launcher-args="--throughput-mode" -b 128 --metrics throughputs
mv .userbenchmark/cpu qat
cat qat/metric* # to see the results
```
Suspected guilty commit: https://github.com/pytorch/pytorch/commit/51a56efbb91377237eb1b81d72c7d598ad61b14e
[torchbench-shufflenet_v2_x1_0-inference-qat-performance-drop_guilty_commit.log](https://github.com/pytorch/pytorch/files/15104655/torchbench-shufflenet_v2_x1_0-inference-qat-performance-drop_guilty_commit.log)
cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @WeizhuoZhang-intel @chuanqi129