Repository: huggingface/lighteval
Issue: [BUG] IFEval metrics incorrect for reasoning models
Root cause: Validation Gap

## Describe the bug

The IFeval scores for reasoning models are significantly worse than their non-reasoning counterparts. Here's an example from the Qwen3 and Phi-4-mini models:

![Image](https://github.com/user-attachments/assets/cc6193bf-e885-414b-932b-059c91237d9f)

The reason for this is due to the fact that reasoning models produce a long CoT, typically enclosed in `<think>` and `</think>` tags. This CoT violates many of the constraints expected in the IFEval prompts. Here's an example from `Qwen3-0.6B` with `/think` mode activated:

**Prompt**
```
Write an extremely short essay on the role of mythology in the work of Jordan Peterson. Keep your entire response 100 words or less. Be general in your writing. Make sure to highlight at least 2 sections in your answer with markdown, i.e. use *highlighted section*.
```

**Completion**
```
<think>
Okay, so I need to write an extremely short essay about Jordan Peterson and mythology. The user wants it under 100 words, with at least two highlighted sections. Let me start by recalling what I know about Jordan Peterson. He's a motivational speaker, known for his "The Secret" book and the "The Integrity of the Self" series. Mythology is a big part of his work, especially in his philosophy.

First, I should mention the connection between his philosophy and mythology. Maybe start with how he uses mythology to explore themes like self-discovery and personal growth. Then, another highlight could be his use of historical figures and their stories to illustrate his ideas. Wait, the user said to highlight two sections. Maybe the first is his approach to personal development, and the second is his use of myths to explain moral and ethical principles.

I need to keep it concise. Let me check the word count. Each section should be a paragraph or two. Make sure the highlighted sections are in markdown. Alright, putting it all together now.
</think>

Jordan Peterson draws on mythology to explore self-discovery and moral integrity, using historical figures like Socrates and Confucius to illustrate philosophical principles. His work emphasizes personal growth and ethical living, blending mythic themes with practical advice. *Highlight section: Personal development and ethical principles.* *Highlight section: Mythic figures and moral teachings.*  

(100 words)
```

Here one sees that the final answer satisfies the instruction, but the thinking block violates the constraint to keep it under 100 words. The solution is to split the completions on `</think>` and only compute metrics on the answer.

## To Reproduce

Run `ifeval` as follows:

```
export VLLM_WORKER_MULTIPROC_METHOD=spawn # Required for vLLM
MODEL=Qwen/Qwen3-0.6B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "extended|ifeval|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

## Expected behavior

The think section of a model completion is excluded from the metrics computation.

## Version info

```
- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.31
- Python version: 3.11.11
- TRL version: 0.19.0.dev0+b093852
- PyTorch version: 2.6.0
- accelerator(s): cpu
- Transformers version: 4.52.4
- Accelerate version: 1.3.0
- Accelerate config: not found
- Datasets version: 3.6.0
- HF Hub version: 0.32.3
- bitsandbytes version: 0.46.0
- DeepSpeed version: 0.16.9
- Diffusers version: 0.33.1
- Liger-Kernel version: 0.5.10
- LLM-Blender version: 0.0.2
- OpenAI version: 1.55.2
- PEFT version: 0.15.2
- vLLM version: 0.8.5.post1
```


Closed by https://github.com/huggingface/lighteval/pull/791

@lewtun #791 has not been included in any LightEval release yet.

We added a general tag for all benchmarks instead on focusing on ifeval.
https://huggingface.co/docs/lighteval/v0.12.2/en/quicktour#general-information