Repository: pytorch/benchmark
Issue: TorchInductor causes performance regression for yolov3 and hf_reformer
Root cause: Algorithmic Error

As the title described, yolov3 and hf_reformer has performance regressions when use torchinductor as the backend compiler.
The commands to run with TorchInductor,
```
python3 run.py -d cuda -t eval  yolov3  --torchdynamo inductor
```
yolov3 has about 30% slowdown. hf_reformer's execution time is increased from 27ms to 6333.45ms.

Do you see the same performance issue using the torchbench.py script here? https://github.com/pytorch/pytorch/blob/master/benchmarks/dynamo/torchbench.py'

Their dashboard: https://github.com/pytorch/pytorch/issues/93794

Is this still the case? @FindHao 

@xuzhao9 It's even worse. Here is [a comparison of the results](https://docs.google.com/spreadsheets/d/1KYJPW12CRG65YgmRfq6uy8B4DPOnNqQimNFxa3qiV10/edit?usp=sharing). The sheet pt2_mar31_train means training tests of TorchBench with PyTorch 2 Mar 31 nightly version, and pt2_train means training tests with PyTorch2 stable version(Mar 9). I have highlighted the regression cases. Some cases have errors when running with pt2_mar31, such as yolov3.

The memory usage is reduced, but it is more related to pytorch changes in my tests.


The tests for pt2_mar31 are done after [the previous commit to fix memory usage](https://github.com/pytorch/benchmark/commit/ea7b71ead75529529d67ffd17541b1f203c49b83). But I manually tested several cases before this commit with pt2_mar31, and the GPU memory usage is already reduced. So I believe the memory usage changes come from PyTorch rather than our measurements.

BTW, we measure [torch._dynamo.optimize](https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/backends/torchdynamo.py#L64) while the benchmark in torchdynamo measures [torch._dynamo.run](https://github.com/pytorch/pytorch/blob/master/benchmarks/dynamo/common.py#L612). According to [the doc](https://pytorch.org/docs/stable/_dynamo.html), `torch._dynamo.run` doesn't do any dynamic compiles, just use prior optimizations. I remember @desertfire mentioned they need further discussions about it. 

Is this issue still relevent?  @FindHao 

For example, I don't see hf_Reformer has performance issue in inductor dashboard:

![image](https://github.com/pytorch/benchmark/assets/502017/c4b17ea4-d24c-4317-8036-2a563ff7884d)

Link: https://hud.pytorch.org/benchmark/torchbench/inductor_with_cudagraphs?startTime=Wed%2C+19+Jul+2023+17%3A52%3A36+GMT&stopTime=Wed%2C+26+Jul+2023+17%3A52%3A36+GMT&granularity=hour&mode=training&dtype=amp&lBranch=main&lCommit=1032a2541ef6b4ddd6090b76b7bab37b24f14676&rBranch=main&rCommit=8daed86e4e441ac5faa330f5636833d8d9e176aa

when I run the command `python3 run.py -d cuda -t eval  yolov3  --torchdynamo inductor`, there is an error listed in the below. I assume it is caused by how torchbench use inductor. But yes, according to inductor dashboard, there is no performance issues in those models. Let's close it.
```
Traceback (most recent call last):
  File "/scratch/yhao24/p9_inductor/benchmark/run.py", line 369, in <module>
    run_one_step(test, model=m, export_metrics_file=export_metrics_file,
  File "/scratch/yhao24/p9_inductor/benchmark/run.py", line 103, in run_one_step
    func()
  File "/scratch/yhao24/p9_inductor/benchmark/torchbenchmark/util/model.py", line 271, in invoke
    out = self.eval()
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_dynamo/eval_frame.py", line 295, in _fn
    return fn(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/benchmark/torchbenchmark/models/yolov3/__init__.py", line 94, in eval
    out = model(*example_inputs, augment=False)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/nn/modules/module.py", line 1514, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/benchmark/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward
    return self.forward_once(x)
  File "/scratch/yhao24/p9_inductor/benchmark/torchbenchmark/models/yolov3/yolo_models.py", line 288, in forward_once
    x = module(x, out)  # WeightedFeatureFusion(), FeatureConcat()
  File "/scratch/yhao24/p9_inductor/pytorch/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/nn/modules/module.py", line 1514, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/benchmark/torchbenchmark/models/yolov3/yolo_utils/layers.py", line 47, in forward
    def forward(self, x, outputs):
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_dynamo/eval_frame.py", line 295, in _fn
    return fn(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_dynamo/external_utils.py", line 17, in inner
    return fn(*args, **kwargs)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_functorch/aot_autograd.py", line 3729, in forward
    return compiled_fn(full_args)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_functorch/aot_autograd.py", line 1433, in g
    return f(*args)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_functorch/aot_autograd.py", line 2385, in runtime_wrapper
    all_outs = call_func_with_args(
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_functorch/aot_autograd.py", line 1457, in call_func_with_args
    out = normalize_as_list(f(args))
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_functorch/aot_autograd.py", line 1433, in g
    return f(*args)
  File "/scratch/yhao24/p9_inductor/pytorch/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/scratch/yhao24/p9_inductor/pytorch/torch/_functorch/aot_autograd.py", line 2841, in forward
    assert all(
AssertionError
```