Repository: modelscope/evalscope
Issue: Why is the performance of the qwen3-4b model worse than that of qwen2.5-7b?

Why is the performance of the qwen3-4b model worse than that of qwen2.5-7b?

I used the Ollama framework to compare the performance of the qwen3-4b and qwen2.5-7b models and found that the time taken for the first token by qwen2.5-7b was better. Is this reasonable?

```
# ollama run llm
ollama serve && ollama run qwen3:4b

# benchmark
evalscope perf \
   --parallel 1 \
   --model qwen3:4b \
   --url http://localhost:11434/v1/chat/completions \
   --api openai \
   --dataset random \
   --min-tokens 1024 \
   --max-tokens 1024 \
   --prefix-length 64 \
   --min-prompt-length 1024 \
   --max-prompt-length 2048 \
   --number 512 \
   --tokenizer-path /data/models/Qwen/Qwen3-4B 
```

Performance results comparison：

![Image](https://github.com/user-attachments/assets/ce0fff3c-1b88-4dbd-88a2-69ac73e1f52f)

![Image](https://github.com/user-attachments/assets/29f6b138-77fb-4b5c-9dd7-615bc6f2bab3)



Ensure both models are tested with identical quantization settings and prompt length (set max-prompt-length 1024), and try higher concurrency.

> Ensure both models are tested with identical quantization settings and prompt length (set max-prompt-length 1024), and try higher concurrency.

https://modelscope.cn/models/Qwen/Qwen3-4B
https://modelscope.cn/models/Qwen/Qwen2.5-7B-Instruct

Currently, both models have bf16 precision, no quantization is performed, and the stress test parameters are the same！

Please set the max prompt length to 1024, ensure there is no variation in prompt length.
```bash
 --min-prompt-length 1024
 --max-prompt-length 1024
```

> > Ensure both models are tested with identical quantization settings and prompt length (set max-prompt-length 1024), and try higher concurrency.
> 
> https://modelscope.cn/models/Qwen/Qwen3-4B https://modelscope.cn/models/Qwen/Qwen2.5-7B-Instruct
> 
> Currently, both models have bf16 precision, no quantization is performed, and the stress test parameters are the same！

from the original descripton, it looks lika ollama (aka Q4 quantization by default) is used?

# ollama run llm
ollama serve && ollama run qwen3:4b

> > > Ensure both models are tested with identical quantization settings and prompt length (set max-prompt-length 1024), and try higher concurrency.
> > 
> > 
> > https://modelscope.cn/models/Qwen/Qwen3-4B https://modelscope.cn/models/Qwen/Qwen2.5-7B-Instruct
> > Currently, both models have bf16 precision, no quantization is performed, and the stress test parameters are the same！
> 
> from the original descripton, it looks lika ollama (aka Q4 quantization by default) is used?
> 
> # ollama run llm
> ollama serve && ollama run qwen3:4b

Yes, currently the default Q4 precision inference is used, but I tried to use the bf16 model and found that Ollama does not currently support the Q3 series model.

![Image](https://github.com/user-attachments/assets/3974ec4a-751a-4fa6-92e8-045b3a4a694d)

> Please set the max prompt length to 1024, ensure there is no variation in prompt length.
> 
>  --min-prompt-length 1024
>  --max-prompt-length 1024

The result is the same！

1) Ollama engine has weak concurrent performance, which is specifically reflected in the output token throughput!
2) Performance comparison:
a. First token time consumption, qwen3-4b is about 1.1 times that of qwen2.5-7b!
b. Output token throughput, qwen3-4b is about 1.54 times that of qwen2.5-7b!
c. Total token throughput, qwen3-4b is about 3 times that of qwen2.5-7b!

![Image](https://github.com/user-attachments/assets/9c6fc70e-5e29-4a4e-98e8-e0b2d5d1d7f2)

![Image](https://github.com/user-attachments/assets/3bcb76f2-d51e-4fb7-a8eb-dc9638c22987)

![Image](https://github.com/user-attachments/assets/84423e61-004b-464e-8a0f-7bb215f229f5)

I think this is related to the Ollama engine, because the performance indicators I used when using the sglang engine are not like this！

i am not sure what we are trying to solve here ...

 technically performance is only evaluated against specific inference engines. 

from the disccusions above, the perf eval is accurate given the inference engines of choices.

> i am not sure what we are trying to solve here ...
> 
> technically performance is only evaluated against specific inference engines.
> 
> from the disccusions above, the perf eval is accurate given the inference engines of choices.

Yes, the background is that we used the ollama and sglang engines and encountered that the performance of qwen3-4b was weaker than that of qwen2.5-7b. Then my prior knowledge was that qwen3-4b should be faster! So I checked it bit by bit. At present, the ollama engine has located that the difference is caused by its pre-filling ability. Could it be caused by the deep thinking ability of qwen3? Therefore, from the perspective of weight size, qwen3-4b has more advantages!

> Thank you for your feedback! We will close this issue now. If you have any further questions, please feel free to reopen it. If EvalScope has been helpful to you, please consider giving us a STAR to show your support. Thank you!