Repository: EleutherAI/lm-evaluation-harness
Issue: Low IFEval in Qwen3
Root cause: Configuration Error

I am getting very low IFeval score compared to their [paper](https://arxiv.org/abs/2505.09388)

Following is the cmd I have used (followed [this](https://github.com/EleutherAI/lm-evaluation-harness/issues/3118#issuecomment-3069732745) and [this](https://github.com/EleutherAI/lm-evaluation-harness/issues/2941#issuecomment-2870733949) with no luck)

```
python -m lm_eval --model vllm \
    --model_args "pretrained=Qwen/Qwen3-32B,max_model_len=8192,data_parallel_size=8,enable_thinking=False" \
    --tasks ifeval \
    --batch_size auto \
    --write_out \
    --show_config \
    --seed 42 \
    --output_path out/qwen3-32B.json \
    --system_instruction "You are a helpful assistant. /no_think.
```

Following is the scores I got:
```
"results": {
  "ifeval": {
    "alias": "ifeval",
    "prompt_level_strict_acc,none": 0.2846580406654344,
    "prompt_level_strict_acc_stderr,none": 0.0194187691064861,
    "inst_level_strict_acc,none": 0.4352517985611511,
    "inst_level_strict_acc_stderr,none": "N/A",
    "prompt_level_loose_acc,none": 0.3290203327171904,
    "prompt_level_loose_acc_stderr,none": 0.02021944211853842,
    "inst_level_loose_acc,none": 0.4712230215827338,
    "inst_level_loose_acc_stderr,none": "N/A"
  }
}
```

Did anyone face this issue? And how to solve that, I 

What scores do you get for IFEval under the `hf` backend rather than `vllm`?

> What scores do you get for IFEval under the `hf` backend rather than `vllm`?

Its not that better. 
```
"results": {
  "ifeval": {
    "alias": "ifeval",
    "prompt_level_strict_acc,none": 0.266173752310536,
    "prompt_level_strict_acc_stderr,none": 0.019018766847290668,
    "inst_level_strict_acc,none": 0.4196642685851319,
    "inst_level_strict_acc_stderr,none": "N/A",
    "prompt_level_loose_acc,none": 0.3068391866913124,
    "prompt_level_loose_acc_stderr,none": 0.019846116338147143,
    "inst_level_loose_acc,none": 0.45323741007194246,
    "inst_level_loose_acc_stderr,none": "N/A"
  }
}
```

The cmd:
```
accelerate launch -m lm_eval --model hf \
    --model_args "pretrained=Qwen/Qwen3-32B" \
    --tasks ifeval \
    --batch_size auto \
    --write_out \
    --show_config \
    --seed 42 \
    --log_samples \
    --output_path out/qwen3_32b \
    --system_instruction "You are a helpful assistant. /no_think."
```

I see you are missing some import switches from the hf model command line: `--apply_chat_template` and `--fewshot_as_multiturn` (although I don't think the fewshot one has an affect on IFEval because I don't think it's a few shot task). This ensures that the chat template is actually applied, and Qwen3 really needs the template for the model to work properly.

Additionally, the Qwen documentation recommends the following when thinking mode is disabled:
> For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-32B#best-practices) section.

Finally, which versions of transformers, lm_eval, pytorch and vllm are you running in your environment? That'll help us narrow down any issues.

> I see you are missing some import switches from the hf model command line: `--apply_chat_template` and `--fewshot_as_multiturn` (although I don't think the fewshot one has an affect on IFEval because I don't think it's a few shot task). This ensures that the chat template is actually applied, and Qwen3 really needs the template for the model to work properly.
> 
> Additionally, the Qwen documentation recommends the following when thinking mode is disabled:
> 
> > For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-32B#best-practices) section.
> 
> Finally, which versions of transformers, lm_eval, pytorch and vllm are you running in your environment? That'll help us narrow down any issues.

```
"results": {
  "ifeval": {
    "alias": "ifeval",
    "prompt_level_strict_acc,none": 0.634011090573013,
    "prompt_level_strict_acc_stderr,none": 0.02072934168450657,
    "inst_level_strict_acc,none": 0.7458033573141487,
    "inst_level_strict_acc_stderr,none": "N/A",
    "prompt_level_loose_acc,none": 0.6469500924214417,
    "prompt_level_loose_acc_stderr,none": 0.020566318668824704,
    "inst_level_loose_acc,none": 0.7565947242206235,
    "inst_level_loose_acc_stderr,none": "N/A"
  }
}
```

This is the final output i get by using the gen_kwargs and apply-chat-template, its still 63 vs 83 that the [paper](https://[arxiv.org/pdf/2505.09388](https://arxiv.org/pdf/2505.09388)) (table 14) has reported

### Versions
Transformers = 4.53.2
lm-eval = 0.4.9
torch = 2.7.0
vllm = 0.9.2


### Final command:
```
accelerate launch -m lm_eval --model hf \
    --model_args "pretrained=Qwen/Qwen3-32B" \
    --tasks ifeval \
    --batch_size auto \
    --write_out \
    --show_config \
    --seed 42 \
    --log_samples \
    --output_path out/hf \
    --cache_requests refresh \
    --gen_kwargs '{"temperature":0.7,"top_p":0.8,"top_k":20,"min_p":0.0}' \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --system_instruction "You are a helpful assistant. /no_think."
```

True, not as good, but we are getting a lot better now! The problem is I think we might need to use thinking to get the full performance back, but that would technically produce outputs that don't follow the instructions because all the content within the thinking tags would violate the instruction rules... so short of modifying the task to strip out thinking content before evaluation I'm not sure what can be done.

cc @baberabb we should consider also adding an option to strip thinking from completed generations too.

> True, not as good, but we are getting a lot better now! The problem is I think we might need to use thinking to get the full performance back, but that would technically produce outputs that don't follow the instructions because all the content within the thinking tags would violate the instruction rules... so short of modifying the task to strip out thinking content before evaluation I'm not sure what can be done.
> 
> cc [@baberabb](https://github.com/baberabb) we should consider also adding an option to strip thinking from completed generations too.

Not really, their table 14 shows the non-thinking models eval, which is 83.2 on ifeval_prompt-strict compared to our's 63 😢 

Stripping the thinking content is critical for evaluating reasoning models like Qwen3 and Deepseek R1. 

any update on this?
I found that lm-eval doesn't strip the thinking token from the start, even if its empty

I tested by setting --model_args  "pretrained=Qwen/Qwen3-8B,enable_thinking=False", the eval score is 0.8299, almost the same to the reported score.

> I tested by setting --model_args "pretrained=Qwen/Qwen3-8B,enable_thinking=False", the eval score is 0.8299, almost the same to the reported score.

can you share your script or the exact command you used? Thanks.
Also, did you check if your generated samples have think token in it?


Doesn't https://github.com/EleutherAI/lm-evaluation-harness/pull/3145 solve this?

```shell
lm_eval --model hf \
    --model_args "pretrained=Qwen/Qwen3-8B,enable_thinking=False" \
    --tasks ifeval \
    --batch_size auto \
    --gen_kwargs '{"temperature":0.7,"top_p":0.8,"top_k":20,"min_p":0.0}' \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --write_out \
    --show_config \
    --seed 42 \
    --log_samples \
    --cache_requests refresh \
    --output_path results/qwen3_8B_hf.json
```

````shell
hf (pretrained=Qwen/Qwen3-8B,enable_thinking=False), gen_kwargs: ({'temperature': 0.7, 'top_p': 0.8, 'top_k': 20, 'min_p': 0.0}), limit: None, num_fewshot: None, batch_size: auto
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      4|none  |     0|inst_level_loose_acc   |↑  |0.8993|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.8741|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.8521|±  |0.0153|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.8133|±  |0.0168|
```