Repository: pytorch/benchmark
Issue: Running huggingface model benchmark
Root cause: Interface Contract Mismatch

with the same PR: https://github.com/pytorch/benchmark/pull/2519

Running `python run_benchmark.py torchao --performance --inference --bfloat16 --inductor-compile-mode max-autotune --huggingface --ci`

Gives error:

```
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 948, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3103, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3231, in inline_call_
    tracer.run()
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1095, in run
    while self.step():
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1007, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1856, in LOAD_ATTR
    self._load_attr(inst)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1846, in _load_attr
    result = BuiltinVariable(getattr).call_function(
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1000, in call_function
    return handler(tx, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 848, in builtin_dispatch
    rv = fn(tx, args, kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 766, in call_self_handler
    result = self_handler(tx, *args, **kwargs)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1716, in call_getattr
    return obj.var_getattr(tx, name)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 244, in var_getattr
    value = self.const_getattr(tx, name)
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/torch/_dynamo/variables/constant.py", line 130, in const_getattr
    member = getattr(self.value, name)
torch._dynamo.exc.InternalTorchDynamoError: AttributeError: 'str' object has no attribute 'size'

from user code:
   File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 978, in forward
    outputs = self.albert(
  File "/home/jerryzh/anaconda3/envs/ao/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 692, in forward
    input_shape = input_ids.size()

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
```

cc @williamwen42 @mlazos if you have time to go through this.

Did a little debugging - error still occurs even with torch.compile disabled (using `set_stance("force_eager")`).

I think the bug is caused here: https://github.com/jerryzh168/benchmark/blob/ec965d54ec52414af8edd5b009da5a64f22fb812/userbenchmark/dynamo/dynamobench/torchao_backend.py#L66 `inputs` is a dict but `benchmark_models` assumes a tuple input (https://github.com/pytorch/ao/blob/ae77f4066fe4966153f769625cba32cbef5cad51/torchao/utils.py#L56). So `inputs` is being unpacked incorrectly.

> Did a little debugging - error still occurs even with torch.compile disabled (using `set_stance("force_eager")`).
> 
> I think the bug is caused here: [jerryzh168/benchmark@`ec965d5`/userbenchmark/dynamo/dynamobench/torchao_backend.py#L66](https://github.com/jerryzh168/benchmark/blob/ec965d54ec52414af8edd5b009da5a64f22fb812/userbenchmark/dynamo/dynamobench/torchao_backend.py#L66) `inputs` is a dict but `benchmark_models` assumes a tuple input ([pytorch/ao@`ae77f40`/torchao/utils.py#L56](https://github.com/pytorch/ao/blob/ae77f4066fe4966153f769625cba32cbef5cad51/torchao/utils.py#L56)). So `inputs` is being unpacked incorrectly.

Ah I see, makes sense, thanks!

verified that this one is because of dictionary input, it's resolved now, thanks for your help! @williamwen42