Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Reference-based evaluation for generation tasks
Root cause: Interface Contract Mismatch

Hi,

Thanks for the great library! I'm currently looking into using `lm-eval` to do some benchmarking and would like to evaluate performance on tasks using reference-based metrics (e.g. BLEU, CHR-F, etc.). It seems as though the translation task hasn't yet been ported into the `big-refactor` branch, so I tried setting up a new task but ran into issues when computing the metrics.

It seems that there's a mismatch between the [metric function's signature](https://github.com/EleutherAI/lm-evaluation-harness/blob/4907defdcc91d4bd091618ccd3cc806304b7048e/lm_eval/api/metrics.py#L221) and the [function call](https://github.com/EleutherAI/lm-evaluation-harness/blob/4907defdcc91d4bd091618ccd3cc806304b7048e/lm_eval/api/task.py#L1016C22-L1016C22), which leads to the following error:

```
Traceback (most recent call last):
  File "lm-evaluation-harness/main.py", line 203, in <module>
    main()
  File "lm-evaluation-harness/main.py", line 155, in main
    results = evaluator.simple_evaluate(
  File "lm-evaluation-harness/lm_eval/utils.py", line 335, in _wrapper
    return fn(*args, **kwargs)
  File "lm-evaluation-harness/lm_eval/evaluator.py", line 137, in simple_evaluate
    results = evaluate(
  File "lm-evaluation-harness/lm_eval/utils.py", line 335, in _wrapper
    return fn(*args, **kwargs)
  File "lm-evaluation-harness/lm_eval/evaluator.py", line 328, in evaluate
    metrics = task.process_results(
  File "lm-evaluation-harness/lm_eval/api/task.py", line 1019, in process_results
    result = self._metric_fn_list[key](
TypeError: bleu() got an unexpected keyword argument 'references'
```

Does the implementation of these metrics differ between the `master` branch and `big-refactor` branch? Sorry if I'm missing something overly basic here, but if you could provide any clarification on this, that'd be much appreciated! 

---

To reproduce the error, here's a basic task definition yaml saved at `lm-evaluation-harness/lm_eval/tasks/translation/wmt16_de_en.yaml`:

```
group:
  - greedy_until
  - wmt16_de_en
task: wmt16_de_en
dataset_path: wmt16
dataset_name: de-en
output_type: greedy_until
training_split: train
validation_split: validation
fewshot_split: validation
test_split: test
doc_to_text: "Translate the following sentence from German to English:\nGerman: {{translation['de']}}\nEnglish:"
doc_to_target: "{{translation['en']}}"
metric_list:
  - metric: bleu
    aggregation: mean
    higher_is_better: true
    ignore_case: false
    ignore_punctuation: false
    regexes_to_ignore:
generation_kwargs:
  until:
    - "\n"
  do_sample: false
  temperature: 0.0
repeats: 1
```

and the run command used for testing:

```bash
python main.py --model hf --model_args pretrained="facebook/opt-350m" --tasks wmt16_de_en --device cuda:0 --write_out --limit 1
```

For reference, my `lm-eval` version is commit `d1a44c852f5973943d25638f0d267c1a5616d999`.

I have a similar issue

Hi, thanks for raising this! I'll check it out in more detail tomorrow. 

This is not intended, sorry. I was planning on looking at porting over WMT this weekend! 

WIP support, which I've tested on my end using the YAML provided, is here: #775 

Will clean this up further and apply the same fix to chrf and ter before merging, but in short: 

```
metric_list:
  - metric: bleu
    aggregation: bleu
    higher_is_better: true
```
or just 
```
metric_list:
   - metric: bleu
```
With no extra kwargs, are the intended usage! This is because bleu scores are calculated on the corpus, not example level, and so the calculation happens technically in the "aggregation" stage.