Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate Qwen3 in non-thinking mode?

As stated in the title, is there a way to add this feature?

Hi! in vllm you can use `enable_thinking=False` to the `model_args` and for HF you'll need to add it [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/2ea6114e17cc5057edb8ea9710e619c43ede2d03/lm_eval/models/huggingface.py#L1441) to the `self.tokenizer.apply_chat_template` call.

Thank you so much! Very helpful advice!

@baberabb 
Sorry, I encounter another question today.
When I test the Qwen3-4B using the following script:
```
model_path="/mnt/gemini/data1/yifengliu/model/Qwen3-4B"
num_gpus=$(echo "$CUDA_VISIBLE_DEVICES" | awk -F',' '{print NF}')


lm_eval --model vllm \
    --model_args pretrained=${model_path},tensor_parallel_size=${num_gpus},dtype="auto",gpu_memory_utilization=0.8,enable_thinking=False \
    --tasks mgsm_direct \
    --output_path /mnt/gemini/data1/yifengliu/qe-lr/output/mgsm \
    --log_samples \
    --batch_size auto
```

The result turns out to be very bad. Its performance is even worse than Qwen2.5-3B-Instruct.
These two models' performance are listed as follows:


  | Bn | De | en | es | fr | ja | ru | sw | te | th | zh | Avg
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Qwen2.5-3B-Instruct | 18.80 | 24.40 | 66.80 | 45.60 | 38.40 | 43.20 | 38.80 | 5.60 | 4.80 | 38.00 | 62.00 | 35.13
Qwen3-4B | 26 | 25.6 | 53.2 | 45.2 | 23.6 | 16.4 | 12.8 | 4 | 15.2 | 20 | 43.6 | 25.96

</google-sheets-html-origin>
PS. The value in the table should be the result of exact_match, I didn't include Stderr into this table.

I'll be more than glad if anyone can help me with it!

> [@baberabb](https://github.com/baberabb) Sorry, I encounter another question today. When I test the Qwen3-4B using the following script:
> 
> ```
> model_path="/mnt/gemini/data1/yifengliu/model/Qwen3-4B"
> num_gpus=$(echo "$CUDA_VISIBLE_DEVICES" | awk -F',' '{print NF}')
> 
> 
> lm_eval --model vllm \
>     --model_args pretrained=${model_path},tensor_parallel_size=${num_gpus},dtype="auto",gpu_memory_utilization=0.8,enable_thinking=False \
>     --tasks mgsm_direct \
>     --output_path /mnt/gemini/data1/yifengliu/qe-lr/output/mgsm \
>     --log_samples \
>     --batch_size auto
> ```
> 
> The result turns out to be very bad. Its performance is even worse than Qwen2.5-3B-Instruct. These two models' performance are listed as follows:
> 
>  	Bn	De	en	es	fr	ja	ru	sw	te	th	zh	Avg
> Qwen2.5-3B-Instruct	18.80	24.40	66.80	45.60	38.40	43.20	38.80	5.60	4.80	38.00	62.00	35.13
> Qwen3-4B	26	25.6	53.2	45.2	23.6	16.4	12.8	4	15.2	20	43.6	25.96
> PS. The value in the table should be the result of exact_match, I didn't include Stderr into this table.
> I'll be more than glad if anyone can help me with it!

I am facing the same issue but for IFEval, the results are quite low compared to what they have reported. Did you find the reason behind this?

> > [@baberabb](https://github.com/baberabb) Sorry, I encounter another question today. When I test the Qwen3-4B using the following script:
> > ```
> > model_path="/mnt/gemini/data1/yifengliu/model/Qwen3-4B"
> > num_gpus=$(echo "$CUDA_VISIBLE_DEVICES" | awk -F',' '{print NF}')
> > 
> > 
> > lm_eval --model vllm \
> >     --model_args pretrained=${model_path},tensor_parallel_size=${num_gpus},dtype="auto",gpu_memory_utilization=0.8,enable_thinking=False \
> >     --tasks mgsm_direct \
> >     --output_path /mnt/gemini/data1/yifengliu/qe-lr/output/mgsm \
> >     --log_samples \
> >     --batch_size auto
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > The result turns out to be very bad. Its performance is even worse than Qwen2.5-3B-Instruct. These two models' performance are listed as follows:
> >  	Bn	De	en	es	fr	ja	ru	sw	te	th	zh	Avg
> > Qwen2.5-3B-Instruct	18.80	24.40	66.80	45.60	38.40	43.20	38.80	5.60	4.80	38.00	62.00	35.13
> > Qwen3-4B	26	25.6	53.2	45.2	23.6	16.4	12.8	4	15.2	20	43.6	25.96
> > PS. The value in the table should be the result of exact_match, I didn't include Stderr into this table.
> > I'll be more than glad if anyone can help me with it!
> 
> I am facing the same issue but for IFEval, the results are quite low compared to what they have reported. Did you find the reason behind this?

Unfortunately, I still didin't find the cause. But, I assume it might relate to answer extraction issues. But lm_evals code is a little bit complex, I still don't have time to check it :(

Hi, do we have any update on this issue or problem? Is there any existing workaround for this?

> Hi, do we have any update on this issue or problem? Is there any existing workaround for this?

Hi! Have you tried with `--apply_chat_template`. Also they recommend sampling params you can add as well: `--gen_kwargs temperature=0.7,top_p=0.8,top_k=20,min_p=0,do_sample=True`

Thank you very much for your reply!
Yes, I think I've tried both `--apply_chat_template` and `--genkwargs`, but the results are still very bad for mgsm_direct
My script is attached below, thank you so much if you could help me figure this out!
```
lm_eval --model vllm \
    --model_args pretrained=${model_path},tensor_parallel_size=${num_gpus},dtype="auto",gpu_memory_utilization=0.8,enable_thinking=False \
    --tasks mgsm_direct \
    --gen_kwargs temperature=0.7,top_p=0.8,top_k=20,min_p=0,do_sample=True \
    --output_path /mnt/gemini/data1/yifengliu/qe-lr/output/mmlu_pro \
    --log_samples \
    --apply_chat_template \
    --batch_size auto