Repository: huggingface/lighteval
Issue: MMLU evaluation fails with Mistral
Root cause: Validation Gap

Evaluating any Mistral checkpoint on MMLU throws an error, suggesting that there are several tokens tokens when only one is permitted:

```
accelerate launch --num_processes=1 run_evals_accelerate.py \
    --model_args "pretrained=hf-internal-testing/tiny-random-MistralForCausalLM" \
    --tasks "original|mmlu|0|0" \
    --override_batch_size 8 \
    --output_dir "./"
```

Traceback:
```
Traceback (most recent call last):
  File "/home/sanchit/mistral-evals/run_evals_accelerate.py", line 82, in <module>
    main(args)
  File "/home/sanchit/lighteval/src/lighteval/logging/hierarchical_logger.py", line 166, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/sanchit/lighteval/src/lighteval/main_accelerate.py", line 111, in main
    evaluation_tracker = evaluate(
                         ^^^^^^^^^
  File "/home/sanchit/lighteval/src/lighteval/evaluator.py", line 84, in evaluate
    full_resps = lm.loglikelihood_single_token(requests, override_bs=override_bs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sanchit/lighteval/src/lighteval/models/base_model.py", line 948, in loglikelihood_single_token
    raise ValueError(
ValueError: Trying to do single token multiple choice but one choice has several tokens: [[28705, 330], [28705, 365], [28705, 334], [28705, 384]]. If the additional pre-token is a space, try to set --no_multichoice_continuations_start_space 
Traceback (most recent call last):
  File "/home/sanchit/miniconda3/envs/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
    args.func(args)
  File "/home/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1075, in launch_command
    simple_launcher(args)
  File "/home/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 681, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/sanchit/miniconda3/envs/venv/bin/python', 'run_evals_accelerate.py', '--model_args', 'pretrained=hf-internal-testing/tiny-random-MistralForCausalLM', '--tasks', './examples/tasks/subset_cosmo_tasks.txt', '--override_batch_size', '8', '--output_dir', './']' returned non-zero exit status 1.
```

Adding the suggested flag `--no_multichoice_continuations_start_space` does not fix the error. For reference: switching to the full `mistralai/Mistral-7B-v0.1` checkpoint also yields the same error.

Env:
- `transformers` version: 4.40.0.dev0
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
- Python version: 3.11.8
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.2
- Lighteval version: 0.4.0.dev0

cc @NathanHB @clefourrier 

Thanks for the report, seems like `--no_multichoice_continuations_start_space` is broken. 

Do you have a preference for using `original|mmlu` over `helm|mmlu` or `leaderboard|mmlu` for example?

If yes, you can likely run it anyway by changing
```
{"name":"mmlu","suite":["original"],"prompt_function":"mmlu_helm","hf_repo":"lighteval\/mmlu","hf_subset":"all","hf_avail_splits":["auxiliary_train","test","validation","dev"],"evaluation_splits":["test"],"few_shots_split":"dev","few_shots_select":"sequential","generation_size":5,"metric":["loglikelihood_acc_single_token"],"stop_sequence":["\n"],"output_regex":null,"frozen":false, "trust_dataset": true}
```
to
```
{"name":"mmlu","suite":["original"],"prompt_function":"mmlu_helm","hf_repo":"lighteval\/mmlu","hf_subset":"all","hf_avail_splits":["auxiliary_train","test","validation","dev"],"evaluation_splits":["test"],"few_shots_split":"dev","few_shots_select":"sequential","generation_size":5,"metric":["loglikelihood_acc"],"stop_sequence":["\n"],"output_regex":null,"frozen":false, "trust_dataset": true}
```
[here](https://github.com/huggingface/lighteval/blob/6a48e4e7723611edda9f51d0fcb9c4d20b79b19d/src/lighteval/tasks/tasks_table.jsonl#L576C1-L576C407) in your file.

(In the meantime, we'll need to investigate)

Thanks @clefourrier! Would there be a difference between the different MMLU configs? I'm looking to reproduce these benchmarks: https://huggingface.co/HuggingFaceTB/cosmo-1b#evaluation

Unless specified, would MMLU take a default form?

- original is the implementation closest to the authors's one, but not a lot of people are actually using it
- helm follows the implementation of stanford (= generative evaluation of mmlu, even less used)
- leaderboard is the closest to the harness, I'd go for this one

Super! Thanks for the help!

Thanks a lot for reporting, found the bug, it should be fixed in the above PR!

Thanks!