Repository: EleutherAI/lm-evaluation-harness
Issue: add `instruct_format` (or `chat_overload` or similar) field to task configs
Root cause: Unimplemented Feature Gap

Users have been reporting that their evaluation scores don't match official benchmarks for awhile (e.g #3407, #3411, #2774
#3405,#2555, #2707, among others) particularly when using instruction-tuned models. The evaluation harness was originally designed primarily for base models, which are evaluated with few-shot examples and rely on in-context pattern recognition (or alternatively multiple-choice/loglikelihoods based evals) . We've  also preferred following the official implementation for each task (where possible), which works well for base models since they behave relatively consistently across different model families (give or take).

However, instruction-tuned models need to be handled differently:

- They're trained to follow explicit instructions and perform significantly better when given clear formatting directives (e.g., "Write your answer as: The answer is [X]")
- Unlike base models, there's no "official" instruction format we can follow as a standard
- Different model families/types may require different instruction formats (e.g thinking models vs vanilla)

#### Proposed Solution
Add a new optional field `chat_overload` to task configurations that provides instruction-specific formatting when a model's `chat_template` is being used.

#### How it works:

- When a model has a chat template defined (either auto-detected or specified), the chat_overload instructions are automatically incorporated
- Base models without chat templates continue using the standard few-shot evaluation format
- This allows each model type to be evaluated properly without requiring manual prompt modifications
- An alternative could be to keep creating different config variants (gsm8k, gsm8k_chat_cot, gsm8k_chat_qwen, ..), but this would get unwieldy fast and I think also confusing to users.

This approach would allow base model evaluations to remain fairly consistent while properly supporting the instruction-tuned models. This would also allow us to support multiple instruction variants as the community iterates.

@Helw150 Curious if you have any thoughts on this or how y'll are handling the base vs instruct variants

@baberabb In my experience, setting `think_end_token` and `--gen_kwargs max_gen_toks` is critical to get good metrics for thinking models & generative tasks. `lm-evaluation-harness` is not strict enough / verbose enough regarding these being set.

For example, for some tasks, it happens that the model does not have enough budget to generation an answer after thinking.

Silently as well, if `think_end_token` is not set, thinking tokens are considered as part of the answer to apply a regex on, which is not great.

I think `lm-evaluation-harness` should be stricter / more verbose regarding these. For example, logging suspicious sequences (finish due to `stop` token like `\n`, finish due to `max_gen_toks` being hit, regex hitting `[invalid]`).

I also believe we not should allow to run `enable_thinking=True` along tasks that use `output_type: multiple_choice` (as https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mmlu/default/_default_template_yaml or https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/piqa/piqa.yaml), as it makes no sense.

WDYT

> [@Helw150](https://github.com/Helw150) Curious if you have any thoughts on this or how y'll are handling the base vs instruct variants

For our own models, we are mostly handling Instruct variants the same as Base variants (just with the apply_chat_template enabled). Admittedly, as @fxmarty-amd is suggesting, we seem to be hitting issue with this as our "Instruct" variants converge towards thinking/reasoning variants.