Repository: pytorch/benchmark
Issue: hf_T5 + DDP fails
Root cause: Interface Contract Mismatch

1. When preparing a DDP model, `self.get_module()` gets called
2. In some huggingface models (including T5, Bart, ... and all the `AutoModelForSeq2SeqLM` models), implementation of `get_module` for huggingface will wrap the model in `ArgsToKwargsWrapper`. This model wrapper only has `input_ids, decoder_input_ids` arguments
3. huggingface `example_inputs` contains a kwarg `labels`
4. When we try to call `forward(**example_inputs)` the call fails because there's no arg or kwarg for `labels`.

Repro:
```
python run_benchmark.py distributed --ngpus 8 --nodes 2 --model torchbenchmark.models.hf_T5.Model --trainer torchbenchmark.util.distributed.core_model.trainer.Trainer --distributed ddp --job_dir [some_directory]
```

Error:
```
$ python run_benchmark.py distributed --ngpus 8 --nodes 2 --model torchbenchmark.models.hf_T5_large.Model --trainer torchbenchmark.util.distributed.core_model.trainer.Trainer
 --distributed ddp --job_dir /fsx/users/dberard/scratch-local/bench-fast/benchmark/native_runner --partition scavenge
/fsx/users/dberard/scratch-local/bench-fast/benchmark/native_runner/bbe0fa8f8a6e41ca9a8923e7c70b6577_init
Traceback (most recent call last):
  File "run_benchmark.py", line 26, in <module>
    run()
  File "run_benchmark.py", line 20, in run
    benchmark.run(bm_args)
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/userbenchmark/distributed/__init__.py", line 46, in run
    result = job.results()
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/core.py", line 287, in results
    return [tp.cast(R, sub_job.result()) for sub_job in self._sub_jobs]
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/core.py", line 287, in <listcomp>
    return [tp.cast(R, sub_job.result()) for sub_job in self._sub_jobs]
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/core.py", line 266, in result
    r = self.results()
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/core.py", line 294, in results
    raise job_exception  # pylint: disable=raising-bad-type
submitit.core.utils.FailedJobError: Job (task=0) failed during processing with trace:
----------------------
Traceback (most recent call last):
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/submission.py", line 54, in process_job
    result = delayed.result()
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/utils.py", line 133, in result
    self._result = self.function(*self.args, **self.kwargs)
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/distributed/submit.py", line 123, in __call__
    return trainer_class(self.args, model_class, model_args=self.model_args).measure()
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/distributed/core_model/trainer.py", line 79, in measure
    self.benchmark.invoke()
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/model.py", line 188, in invoke
    self.train()
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 115, in train
    outputs = self.model(**self.example_inputs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'labels'

----------------------
You can check full logs with 'job.stderr(0)' and 'job.stdout(0)'or at paths:
  - /fsx/users/dberard/scratch-local/bench-fast/benchmark/native_runner/63035_0_log.err
  - /fsx/users/dberard/scratch-local/bench-fast/benchmark/native_runner/63035_0_log.out

Repro-ing

I can't immediately reproduce this. Maybe we have different versions of the transformers library?

I'm running this command ```python run_benchmark.py distributed --ngpus 8 --nodes 1  --model torchbenchmark.e2e_models.hf_t5.Model --trainer torchbenchmark.util.distributed.trainer.Trainer --distributed ddp --job_dir $PWD/.userbenchmark/distributed/e2e_hf_t5```

Edit: We're using different run commands. I'm not sure if running distributed and torchbenchmark.models.hf_t5 is supposed to compose. It has to be the one from e2e_models. 

Edit2: It does seem like a worthy fix to have DDP and regular hf_t5 compose though. I'm not sure why they don't. Is models.hf_t5 just broken in general? hm. 

I can reproduce this problem on AWS cluster. Working on the fix.