Repository: EleutherAI/lm-evaluation-harness
Issue: Allow `gen_prefix` after thinking / clarify that it is not compatible with `enable_thinking=True`
Root cause: Architectural Constraint

Hi,

Currently, `gen_prefix` prevents models to use think tokens. This may result in worse evaluations, as some models are not intended to be used without think tokens (e.g. gpt-oss AFAIK).

For example, with https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/arc/arc_challenge_chat.yaml, the prompt looks like:

`'
system
You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\nCurrent date: 2025-11-13\n\nReasoning: low\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.

user
Given the following question and four candidate answers (A, B, C and D), choose the best answer.\nQuestion: An astronaut drops a 1.0 kg object and a 5.0 kg object on the Moon. Both objects fall a total distance of 2.0 m vertically. Which of the following best describes the objects after they have fallen a distance of 1.0 m?\nA. They have each lost kinetic energy.\nB. They have each gained the same amount of potential energy.\nC. They have each lost the same amount of potential energy.\nD. They have each gained one-half of their maximum kinetic energy.\nYour response should end with "The best answer is [the_answer_letter]" where the [the_answer_letter] is one of A, B, C or D.

assistant
final
The best answer is'`

and the model generates: `' D
'`.

Without `gen_prefix: 'The best answer is'`, the prompt looks like:

`'
system
You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\nCurrent date: 2025-11-13\n\nReasoning: low\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.

user
Given the following question and four candidate answers (A, B, C and D), choose the best answer.\nQuestion: An astronaut drops a 1.0 kg object and a 5.0 kg object on the Moon. Both objects fall a total distance of 2.0 m vertically. Which of the following best describes the objects after they have fallen a distance of 1.0 m?\nA. They have each lost kinetic energy.\nB. They have each gained the same amount of potential energy.\nC. They have each lost the same amount of potential energy.\nD. They have each gained one-half of their maximum kinetic energy.\nYour response should end with "The best answer is [the_answer_letter]" where the [the_answer_letter] is one of A, B, C or D.

assistant'`

and the generated text: `'
analysis
We need after falling 1m out of 2m. Initially at rest. Potential energy loss = m g h. For 1m, potential lost = m g *1. Kinetic gained = same. So each has gained kinetic equal to potential lost. They have each gained the same amount of kinetic energy? Not listed. They have each lost the same amount of potential energy? For 1m, potential lost proportional to mass, so not same. They have each gained one-half of their maximum kinetic energy. Max kinetic at bottom after 2m: KE = m g *2. So at 1m, KE = m g *1 = half of max. So D.

assistant
final
The best answer is D
'`.

In this case, the answer is the same, but there could be cases where thinking helps.

Ideally, to enable thinking along a template start output, we would allow thinking until `
assistant
final
` is generated, add the `gen_prefix`, and the continue generating from there.

Thank you