Repository: pytorch/benchmark
Issue: [train][hf_Whisper][hf_T5_generate] TypeError: forward() got an unexpected keyword argument 'input_ids'
Root cause: Unimplemented Feature Gap

Hi,

I meet the error for argument 'input_ids' when training model with following commands on A100.
```
python -u run.py -d cuda -t train --bs 4 --metrics None hf_Whisper
python -u run.py -d cuda -t eval --bs 4 --metrics None hf_T5_generate
```

Here is the detailed log:
```
Running train method from hf_T5_generate on cuda in eager mode with input batch size 4 and precision fp32.
Traceback (most recent call last):
  File "/home/pt-gpu/zejun/benchmark/run.py", line 627, in <module>
    main()  # pragma: no cover
  File "/home/pt-gpu/zejun/benchmark/run.py", line 597, in main
    run_one_step(
  File "/home/pt-gpu/zejun/benchmark/run.py", line 177, in run_one_step
    func()
  File "/home/pt-gpu/zejun/benchmark/torchbenchmark/util/model.py", line 309, in invoke
    return self._invoke_staged_train_test(num_batch=self.num_batch)
  File "/home/pt-gpu/zejun/benchmark/torchbenchmark/util/model.py", line 299, in _invoke_staged_train_test
    losses = self.forward()
  File "/home/pt-gpu/zejun/benchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 157, in forward
    outputs = self.model(**self.example_inputs)
  File "/home/pt-gpu/anaconda3/envs/zejun/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/pt-gpu/anaconda3/envs/zejun/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'input_ids'
```

Environment:
torch: 2.1.0
torchbench: ec35e3f5da08587249d83248762236d711d19f9b
python: 3.9.17


I tried in our nightly docker, but couldn't reproduce the error:
```
$ python -u run.py -d cuda -t eval --bs 4 --metrics None hf_T5_generate
/home/runner/miniconda3/envs/torchbench/lib/python3.11/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/runner/miniconda3/envs/torchbench/lib/python3.11/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/runner/miniconda3/envs/torchbench/lib/python3.11/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
Running eval method from hf_T5_generate on cuda in eager mode with input batch size 4 and precision fp16.
GPU Time per batch:  3436.564 milliseconds
CPU Wall Time per batch: 3436.605 milliseconds
CPU Wall Time:       3436.605 milliseconds
```

It seems in our latest code, the default precision is fp16, but you are running with fp32:
https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/framework/huggingface/model_factory.py#L66

Can you please try update your torchbench and huggingface transformer package version, or try to reproduce using our docker image? It is available at https://github.com/pytorch/benchmark/pkgs/container/torchbench

Hi, @xuzhao9 

Thank you for your help. I find the error happened only when run training, here is the command:
python -u run.py -d cuda -t train --bs 4 --metrics None hf_Whisper
```
(zejun) pt-gpu@mlp-milan-02:~/zejun/benchmark$ python -u run.py -d cuda -t train --bs 4 --metrics None hf_Whisper
Running train method from hf_Whisper on cuda in eager mode with input batch size 4 and precision fp32.
Traceback (most recent call last):
  File "/home/pt-gpu/zejun/benchmark/run.py", line 623, in <module>
    main()  # pragma: no cover
  File "/home/pt-gpu/zejun/benchmark/run.py", line 593, in main
    run_one_step(
  File "/home/pt-gpu/zejun/benchmark/run.py", line 173, in run_one_step
    func()
  File "/home/pt-gpu/zejun/benchmark/torchbenchmark/util/model.py", line 309, in invoke
    return self._invoke_staged_train_test(num_batch=self.num_batch)
  File "/home/pt-gpu/zejun/benchmark/torchbenchmark/util/model.py", line 299, in _invoke_staged_train_test
    losses = self.forward()
  File "/home/pt-gpu/zejun/benchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 157, in forward
    outputs = self.model(**self.example_inputs)
  File "/home/pt-gpu/anaconda3/envs/zejun/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/pt-gpu/anaconda3/envs/zejun/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'input_ids'
```

When run eval with following command, i get the same log and the model passed.
python -u run.py -d cuda -t eval --bs 4 --metrics None hf_T5_generate
```
(zejun) pt-gpu@mlp-milan-02:~/zejun/benchmark$ python -u run.py -d cuda -t eval --bs 4 --metrics None hf_T5_generate
Running eval method from hf_T5_generate on cuda in eager mode with input batch size 4 and precision fp16.
GPU Time per batch:  2030.264 milliseconds
CPU Wall Time per batch: 2030.285 milliseconds
```

So it looks like the default precision for training of this model is fp32 and the error is only happen in train mode.

I don't think the train test of `hf_Whisper` has been implemented: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/hf_Whisper/__init__.py#L20.
We should throw NotImplementedError instead of printing a stackbacktrace.

Created https://github.com/pytorch/benchmark/pull/2174 to address this issue.

cc the model owner @MaanavD 

Thank you for help @xuzhao9