Repository: EleutherAI/lm-evaluation-harness
Issue: Error when running request `generate_until`
Root cause: Validation Gap

Here is my config

```yaml
group: open_llm_leaderboard
task:
  - task: arc_challenge
    task_alias: arc 25 shot
    num_fewshot: 25
    metric_list:
      - metric: acc_norm
  - task: hellaswag
    task_alias: hellaswag 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: hellaswag_id
    task_alias: hellaswag_id 10 shot
    process_docs: !function hellaswag_utils.process_docs
    num_fewshot: 10
    metric_list:
      - metric: acc_norm
  - task: truthfulqa_mc2
    task_alias: truthfulqa 0 shot
    dataset_path: truthful_qa
    dataset_name: multiple_choice
    output_type: multiple_choice
    process_results: !function truthfulqa_utils.process_results_mc2
    metric_list:
      - metric: acc
  - task: winogrande
    task_alias: winogrande 5 shot
    doc_to_text: !function preprocess_winogrande.doc_to_text
    doc_to_target: !function preprocess_winogrande.doc_to_target
    doc_to_choice: !function preprocess_winogrande.doc_to_choice
    num_fewshot: 5
    metric_list:
      - metric: acc
  - task: gsm8k
    task_alias: gsm8k 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
```

And here is the error

```python
2024-01-17:18:53:16,550 INFO     [evaluator.py:314] Running generate_until requests
  0%|                                                                                                                              | 0/1319 [00:00<?, ?it/s]Passed argument batch_size = auto. Detecting largest batch size
Determined Largest batch size: 8
/home/fahadh/anaconda3/envs/merging/lib/python3.10/site-packages/transformers/generation/utils.py:1518: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use and modify the model generation configuration (see https://huggingface.co/docs/transformers/generation_strategies#default-text-generation-configuration )
  warnings.warn(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [1:10:00<00:00,  3.18s/it]Traceback (most recent call last):
  File "/home/fahadh/anaconda3/envs/merging/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/fahadh/lm-evaluation-harness/lm_eval/__main__.py", line 231, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/fahadh/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/home/fahadh/lm-evaluation-harness/lm_eval/evaluator.py", line 150, in simple_evaluate
    results = evaluate(
  File "/home/fahadh/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/home/fahadh/lm-evaluation-harness/lm_eval/evaluator.py", line 451, in evaluate
    results[task_name][metric_key] = agg_fn(items)
  File "/home/fahadh/lm-evaluation-harness/lm_eval/api/metrics.py", line 20, in mean
    return sum(arr) / len(arr)
TypeError: unsupported operand type(s) for +: 'int' and 'list'
```

EDIT: The problem is from gsm8k task. When I tried to do that task itself, I got the same error. Here is the config

```yaml
group: open_llm_leaderboard
task:
  - task: gsm8k
    task_alias: gsm8k 5 shot
    num_fewshot: 5
    metric_list:
      - metric: acc
```

I have the same problem with `polemo2_in` task.

`gsm8k` is a `generate_until` task and needs to use `exact_match` instead of `acc` which is already set in it's original yaml.

That aside, I've made a fix in #1315 so that you don't have to copy utils functions so you can remove the lines with `!function`

Also, is `hellaswag_id` a custom task you made? I don't think it's in lm-eval currently.

> `gsm8k` is a `generate_until` task and needs to use `exact_match` instead of `acc` which is already set in it's original yaml.
> 
> That aside, I've made a fix in #1315 so that you don't have to copy utils functions so you can remove the lines with `!function`

But, from [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), gsm8k is measured using `acc`. Or did I understand it wrongly?

> Also, is `hellaswag_id` a custom task you made? I don't think it's in lm-eval currently.

it is not? But I got it from `lm_eval --task list`. Maybe because I install this repo including `[multilingual]` optional dependencies?

Yeah, nvm, I forgot where it was located.

Anyway, the Open LLM Leaderboard uses an older version of lm-eval before the big-refactor. Currently, for `generate_until` tasks, we are opting to use `exact_match` as it has some control over minor preprocessing stuff like where or not miss-capitalization should be penalized etc. You should check their implementation to make sure if `acc` is simply string matching or includes some postprocess and maybe adjust the `exact_match` param accordingly. 

> `gsm8k` is a `generate_until` task and needs to use `exact_match` instead of `acc` which is already set in it's original yaml.

I guess polemo2_in can't be exact_match...



polemo2_in has accuracy which I think can also be used for gsm8k and is probably the same as acc. Lm-eval supports using metrics from Huggingface's evaluate library.

So, in short `acc` is not a supported metric for `gsm8k`?

@djstrong could you check if this patch works?
https://github.com/EleutherAI/lm-evaluation-harness/pull/1318

@fahadh4ilyas `acc` is just a code name for accuracy, `gsm8k` uses a variant that is used for string matching but works in a similar sense which is `exact_match` in the current version. 

@lintangsutawika Sorry, it works without the patch :) I haven't checked with the patch.

@fahadh4ilyas Try with `accuracy`.

Perhaps to minimize confusion we should simply add a `name` argument for each metric in the config, that can be used to override what the name of the metric is reported as, so that GSM8k (and perhaps other tasks) can retain the name `acc` for its metric computation.