Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Wrong AutoModel Assignment for MBart
Root cause: Algorithmic Error

I am trying to evaluate some models based on `mbart`. This model is a little bit interesting it can be loaded both with `transformers.AutoModelForCausalLM` and `transformers.AutoModelForSeq2SeqLM` classes. The `mbart` keyword is in both both `MODEL_FOR_CAUSAL_LM_MAPPING_NAMES` and `MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES`.

Because the model class determination first checks if `mbart` is in `MODEL_FOR_CAUSAL_LM_MAPPING_NAMES` list, it loads the model using the `transformers.AutoModelForCausalLM`, however if the model is used to be seq2seq the model wouldn't be used properly.

Is there a way to manually define the models auto class? 

Hi! Trying to understand the issue better—is a suitable fix to simply always use AutoModelForSeq2SeqLM if the model matches MODEL_FOR_SEQ_TO_SEQ_MAPPING_NAMES?

Or are there situations in which you expect one would want to use an mbart model with the causal logic? I’m expecting not, but happy to hear if there is a case for it.

Hi, thank you for your reply. In my case, that simple fix would resolve my issue.

It is most likely if a model matches MODEL_FOR_SEQ_TO_SEQ_MAPPING_NAMES, it would be used with AutoModelForSeq2SeqLM most of the time. And it is less likely to use a seq2seq model with AutoModelForCausalLM. 

However, seems like it is [possible](https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartForCausalLM) to use mbart with AutoModelForCausalLM class even if its a seq2seq model. Unfortunately, I don't know if it is really used by people or not, or I don't know why it is required.



Thank you! I’ll make the above simpler fix then so that these get loaded as Seq2seq properly.

I’m also unsure as to why HF allows these to be used via AutoModelForCausalLM, but seeing as they will still require treating them as Seq2Seq we can ignore that detail.

hi @mcemilg , is there a task for which MBart / a finetuned MBart should be expected to perform well on that I could test? Seeing that its performance on `lambada_openai` is bad before and after this fix, but this is to my knowledge expected because it's not made for english-to-english generation.

I also just recalled that MBart uses some (poorly documented) special language ID tokens, might this be an additional edge case that has to be solved for mbart outside of the fix I've PR'ed?

hi @haileyschoelkopf , thanks for your effort. 

The mbart is commonly used seq2seq multilingual tasks, the most common one is translation. You can evaleate an example mBart model `facebook/mbart-large-50-many-to-many-mmt` using wmt2016. There is already a task there which is for en to ro using t5 model. It can be converted for mbart like this, I tested and it works:


tr_en-mBart.yaml
```
group:
  - wmt-mBart-prompt
task: wmt-tr-en-mBart-prompt
dataset_path: wmt16
dataset_name: tr-en
training_split: train
validation_split: validation
output_type: generate_until
doc_to_text: "{{translation.en}}"
doc_to_target: "{{translation.tr}}"
metric_list:
  - metric: wer
    aggregation: mean
    higher_is_better: false
  - metric: !function metrics.bleu
    aggregation: !function metrics.agg_bleu
    higher_is_better: true
generation_kwargs:
  forced_bos_token_id: 250023
  until: [
    "
"
  ]
```

example usage
```
$ python -m lm_eval --model hf --model_args pretrained=facebook/mbart-large-50-many-to-many-mmt --tasks wmt-tr-en-mBart-prompt --device cuda:0
```


You are right about the special tokens. In the mBart case, you have to provide a special token based on the language you want to generate. It is possible to add a special token to the beginning of the target text using `generation_kwargs`. In the example above, the `forced_bos_token_id` is targeted for Turkish. This works because the source text is in English and the default source language token is in English.

To be able to use another language as a source text, you have to add a special token in the beginning of text of the source. This case is not possible with the existing usage. There should be a way to define special tokens of tokenizers as we did in generation_kwargs. 

This can also be manually done by adding a special token to the `doc_to_text` in yaml if the add_special_tokens=False while encoding with a tokenizer. However, from what I see, the add_psecial_tokens argument is True in default for AutoModelForSeq2SeqLM. This is also another issue. I am not sure but here is the source of issue [L431](https://github.com/EleutherAI/lm-evaluation-harness/blob/b189066ddf970be1ec6d0deec31c121b9b83a4c9/lm_eval/models/huggingface.py#L431) and [L455](https://github.com/EleutherAI/lm-evaluation-harness/blob/b189066ddf970be1ec6d0deec31c121b9b83a4c9/lm_eval/models/huggingface.py#L455) from what I see.







Thanks very much for the thorough documentation of this + MBart-specific usage! I'll follow up on `add_special_tokens=True` potentially not being correct for Enc-dec models.