Repository: pytorch
Issue: [inductor][cpu]transformers models static/dynamic quant performance/accuracy crash in 2024-06-17 nightly release
Labels: oncall: pt2,module: dynamic shapes,oncall: cpu inductor

### 🐛 Describe the bug

```
======================= export model ===============================
W0617 17:40:01.166444 140156037509568 torch/_export/__init__.py:95] +============================+
W0617 17:40:01.166611 140156037509568 torch/_export/__init__.py:96] |     !!!   WARNING   !!!    |
W0617 17:40:01.166671 140156037509568 torch/_export/__init__.py:97] +============================+
W0617 17:40:01.166718 140156037509568 torch/_export/__init__.py:98] capture_pre_autograd_graph() is deprecated and doesn't provide any function guarantee moving forward.
W0617 17:40:01.166769 140156037509568 torch/_export/__init__.py:99] Please switch to use torch.export instead.
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0] Error while creating guard:
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0] Name: ''
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     Source: shape_env
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     Create Function: SHAPE_ENV
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     Guard Types: None
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     Code List: None
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     Object Weakref: None
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     Guarded Class Weakref: None
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0] Traceback (most recent call last):
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]   File "/workspace/pytorch/torch/_guards.py", line 259, in create
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     return self.create_fn(builder, self)
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]   File "/workspace/pytorch/torch/_dynamo/guards.py", line 1728, in SHAPE_ENV
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     guards = output_graph.shape_env.produce_guards(
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]   File "/workspace/pytorch/torch/fx/experimental/symbolic_shapes.py", line 4167, in produce_guards
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]     raise ConstraintViolationError(
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (dim0)! For more information, run with TORCH_LOGS="+dynamic".
E0617 17:40:03.274544 140156037509568 torch/_guards.py:261] [0/0]   - Not all values of dim0 = L['input_ids'].size()[0] in the specified range satisfy the generated guard Ne(L['input_ids'].size()[0], 9223372036854775807).
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0] Created at:
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]   File "/workspace/pytorch/torch/_dynamo/convert_frame.py", line 564, in transform
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]     tracer = InstructionTranslator(
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]   File "/workspace/pytorch/torch/_dynamo/symbolic_convert.py", line 2371, in __init__
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]     output=OutputGraph(
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]   File "/workspace/pytorch/torch/_dynamo/output_graph.py", line 313, in __init__
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]     self.init_ambient_guards()
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]   File "/workspace/pytorch/torch/_dynamo/output_graph.py", line 452, in init_ambient_guards
E0617 17:40:03.275653 140156037509568 torch/_guards.py:263] [0/0]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
Traceback (most recent call last):
  File "./transformers/examples/pytorch/text-classification/run_glue.py", line 652, in <module>
    main()
  File "./transformers/examples/pytorch/text-classification/run_glue.py", line 590, in main
    metrics = trainer.evaluate(eval_dataset=eval_dataset)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 3109, in evaluate
    start_time, output = eval_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 3232, in evaluation_loop
    else self.accelerator.prepare_model(model, evaluation_mode=True)
  File "/opt/conda/lib/python3.8/site-packages/accelerate/accelerator.py", line 1449, in prepare_model
    exported_model = capture_pre_autograd_graph(
  File "/workspace/pytorch/torch/_export/__init__.py", line 170, in capture_pre_autograd_graph
    m = torch._dynamo.export(
  File "/workspace/pytorch/torch/_dynamo/eval_frame.py", line 1425, in inner
    raise constraint_violation_error
  File "/workspace/pytorch/torch/_dynamo/eval_frame.py", line 1379, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1566, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1575, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/pytorch/torch/_dynamo/eval_frame.py", line 433, in _fn
    return fn(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1566, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1575, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/pytorch/torch/_dynamo/convert_frame.py", line 1116, in __call__
    return self._torchdynamo_orig_callable(
  File "/workspace/pytorch/torch/_dynamo/convert_frame.py", line 472, in __call__
    return _compile(
  File "/workspace/pytorch/torch/_utils_internal.py", line 84, in wrapper_function
    return StrobelightCompileTimeProfiler.profile_compile_time(
  File "/workspace/pytorch/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/workspace/pytorch/torch/_dynamo/convert_frame.py", line 817, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/workspace/pytorch/torch/_dynamo/utils.py", line 231, in time_wrapper
    r = func(*args, **kwargs)
  File "/workspace/pytorch/torch/_dynamo/convert_frame.py", line 726, in compile_inner
    check_fn = CheckFunctionManager(
  File "/workspace/pytorch/torch/_dynamo/guards.py", line 2141, in __init__
    guard.create(builder)
  File "/workspace/pytorch/torch/_guards.py", line 259, in create
    return self.create_fn(builder, self)
  File "/workspace/pytorch/torch/_dynamo/guards.py", line 1728, in SHAPE_ENV
    guards = output_graph.shape_env.produce_guards(
  File "/workspace/pytorch/torch/fx/experimental/symbolic_shapes.py", line 4167, in produce_guards
    raise ConstraintViolationError(
torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (dim0)! For more information, run with TORCH_LOGS="+dynamic".
  - Not all values of dim0 = L['input_ids'].size()[0] in the specified range satisfy the generated guard Ne(L['input_ids'].size()[0], 9223372036854775807).

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/workspace/pytorch/numa_launcher.py", line 805, in <module>
    main()
  File "/workspace/pytorch/numa_launcher.py", line 800, in main
    launcher.launch(args)
  File "/workspace/pytorch/numa_launcher.py", line 481, in launch
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd_s)
subprocess.CalledProcessError: Command 'numactl -C 0-31 -m 0 /opt/conda/bin/python -u ./transformers/examples/pytorch/text-classification/run_glue.py --model_name_or_path albert-base-v1 --task_name MRPC --do_eval --max_seq_length 16 --learning_rate 2e-5 --overwrite_output_dir --output_dir /tmp/tmp_huggingface/ --torch_compile --torch_compile_quant ptq_dynamic --report_to=none --per_device_eval_batch_size 64' returned non-zero exit status 1.
```

### Versions

</table><p>SW info</p><table border="1" class="dataframe table">
  <thead>
    <tr style="text-align: right;">
      <th>SW</th>
      <th>Branch</th>
      <th>Target commit</th>
      <th>Refer commit</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Pytorch</td>
      <td>nightly</td>
      <td>8410bf5</td>
      <td>963d450</td>
    </tr>
    <tr>
      <td>Torchbench</td>
      <td>chuanqiw/inductor_quant</td>
      <td>ee35d764</td>
      <td>ee35d764</td>
    </tr>
    <tr>
      <td>torchaudio</td>
      <td>nightly</td>
      <td>b829e93</td>
      <td>1980f8a</td>
    </tr>
    <tr>
      <td>torchtext</td>
      <td>nightly</td>
      <td>b0ebddc</td>
      <td>b0ebddc</td>
    </tr>
    <tr>
      <td>torchvision</td>
      <td>nightly</td>
      <td>d23a6e1</td>
      <td>d23a6e1</td>
    </tr>
    <tr>
      <td>torchdata</td>
      <td>nightly</td>
      <td>11bb5b8</td>
      <td>11bb5b8</td>
    </tr>
    <tr>
      <td>dynamo_benchmarks</td>
      <td>nightly</td>
      <td>fea73cb</td>
      <td>fea73cb</td>
    </tr>
  </tbody>
</table>

</table>

Repro:
```
git clone -b test https://github.com/chuanqi129/transformers && cd transformers && \
    python setup.py bdist_wheel && pip install --force-reinstall dist/*.whl && cd ..
git clone -b test https://github.com/zxd1997066/accelerate.git && cd accelerate && \
    python setup.py bdist_wheel && pip install --no-deps --force-reinstall dist/*.whl && cd ..
pip install -r transformers/examples/pytorch/text-classification/requirements.txt
wget https://github.com/chuanqi129/inductor-tools/raw/xiangdong/accuracy/scripts/modelbench/quant/numa_launcher.py
wget https://github.com/chuanqi129/inductor-tools/raw/xiangdong/accuracy/scripts/modelbench/quant/hf_quant_test.sh
#change model in https://github.com/chuanqi129/inductor-tools/blob/xiangdong/accuracy/scripts/modelbench/quant/hf_quant_test.sh#L88
#static quantization
bash hf_quant_test.sh key torch_compile_quant_static
#dynamic quantization
bash hf_quant_test.sh key torch_compile_quant
```

Suspected guilty commit: https://github.com/pytorch/pytorch/commit/2229884102ac95c9dda0aeadbded1b04295d892e
[text-classification+albert-base-v1-static-quant-accuracy-crash_guilty_commit.log](https://github.com/user-attachments/files/15881130/text-classification%2Balbert-base-v1-static-quant-accuracy-crash_guilty_commit.log)

cc @ezyang @anijain2305 @chauhang @penguinwu @WeizhuoZhang-intel @chuanqi129