Repository: modelscope/evalscope
Issue: Qwen3-4B-Instruct-2507 在 AIME25 数据集上评测结果差异较大
Root cause: Configuration Error

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

在AIME25数据集按照Qwen3技术报告中上以每个样本重复64次的方式，计算平均正确率，得到的结果为43.75，但对应model card中的结果为47.4。两者结果存在明显差异。

## EvalScope 版本（必填）
v1.3.0

## 使用的工具
- [x] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

``` python
from evalscope import TaskConfig, run_task  
  
task_cfg = TaskConfig(  
    model='/usr/weights/qwen3-4b-instruct-2507',  
    api_url='http://127.0.0.1:8000/v1/chat/completions',  
	eval_type='openai_api',
    repeats=64,
	datasets=['aime25'], 
    dataset_args={
        'aime25': {  
            'aggregation': 'mean_and_pass_at_k',  # 使用pass@k聚合方法  
            'few_shot_num': 0  # AIME25通常使用0-shot  
        }  
    }, 
	eval_batch_size=64,
    generation_config={  
        'max_tokens': 16384,  
        'temperature': 0.7,  
        'top_p': 0.8,  
        'top_k': 20,  
        'extra_body':{'chat_template_kwargs': {'enable_thinking': False}}  # 关闭思考模式  
    },  
    timeout=60000,  
    stream=True, 
    limit=None
)  
  
res = run_task(task_cfg=task_cfg)
```
vllm 启动命令如下：

``` sh
vllm serve  /usr/weights/qwen3-4b-instruct-2507 --max-model-len 20000 --enforce-eager --gpu-memory-utilization 0.95
```

## 错误日志

``` text 
2025-12-17 16:48:33 - evalscope - INFO: 
aime25 report table:
+------------------------+-----------+------------------+-------------+-------+---------+---------+
| Model                  | Dataset   | Metric           | Subset      |   Num |   Score | Cat.0   |
+========================+===========+==================+=============+=======+=========+=========+
| qwen3-4b-instruct-2507 | aime25    | mean_acc         | AIME2025-I  |   960 |  0.4323 | default |
+------------------------+-----------+------------------+-------------+-------+---------+---------+
| qwen3-4b-instruct-2507 | aime25    | mean_acc         | AIME2025-II |   960 |  0.4427 | default |
+------------------------+-----------+------------------+-------------+-------+---------+---------+
| qwen3-4b-instruct-2507 | aime25    | mean_acc         | OVERALL     |  1920 |  0.4375 | -       |
```

## 运行环境

- 操作系统：ubuntu 22.04
- Python版本：3.10.12
- vllm版本：0.9.0.1
## 其他信息

如果有其他相关信息，请在此处提供。


generation_config 设置是否与报告中一致

```text 
To achieve optimal performance, we recommend the following settings:

Sampling Parameters:

We suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0.
For supported frameworks, you can adjust the presence_penalty parameter between 0 and 2 to reduce endless repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
``` 
按照Model card 这里的建议进行设置

请先按这里的建议排查一下问题，是模型输出问题还是框架答案解析的问题：https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html#id6

我前段时间跑过原始 qwen3-4b-instruct-2507 的评测，找了一下结果：
```log
+----------+-----------+------------------+-------------+-------+---------+---------+
| Model    | Dataset   | Metric           | Subset      |   Num |   Score | Cat.0   |
+==========+===========+==================+=============+=======+=========+=========+
| qwen3-4b | aime25    | mean_acc         | AIME2025-I  |   960 |  0.4594 | default |
+----------+-----------+------------------+-------------+-------+---------+---------+
| qwen3-4b | aime25    | mean_acc         | AIME2025-II |   960 |  0.4688 | default |
+----------+-----------+------------------+-------------+-------+---------+---------+
| qwen3-4b | aime25    | mean_acc         | OVERALL     |  1920 |  0.4641 | -       |
+----------+-----------+------------------+-------------+-------+---------+---------+
```
平均 acc 是 46.41，比你测的高一些。

相关脚本和参数设置如下：
```sh
# vLLM 启动，2 x A100
# vLLM API server version 0.10.1.1

CUDA_VISIBLE_DEVICES=5,6 TZ="Asia/Shanghai" \
    python -m vllm.entrypoints.openai.api_server \
    --model <model-path> \
    --served-model-name qwen3-4b \
    --tensor-parallel-size 2 \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.97 \
    --trust-remote-code \
    --max-model-len 50000 \
    --port 32104 \
    --max-num-seqs 200 \
    --enable-chunked-prefill --enable-prefix-caching
```

```py
# 评测和生成参数
# evalscope 2.0.0
...
	eval_type="openai_api",
    repeats=64,
	datasets=["aime25"], 
    dataset_args={
        "aime25": {  
            "aggregation": "mean_and_pass_at_k",
            "few_shot_num": 0
        }
    }, 
	eval_batch_size=100,
    generation_config={
        "max_tokens": 38912,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5
    },  
    timeout=60000,
    stream=True, 
...
```
生成参数按 Qwen3 技术报告 P13 的描述：

> For Qwen3 models in the non-thinking mode, we configure the sampling hyperparameters with temperature = 0.7, top-p = 0.8, top-k = 20, and presence penalty = 1.5. For both the thinking and non-thinking modes, we set the max output length to 32,768 tokens, except AIME’24 and AIME’25 where we extend this length to 38,912 tokens to provide sufficient thinking space.



> 我前段时间跑过原始 qwen3-4b-instruct-2507 的评测，找了一下结果：
> 
> ```
> +----------+-----------+------------------+-------------+-------+---------+---------+
> | Model    | Dataset   | Metric           | Subset      |   Num |   Score | Cat.0   |
> +==========+===========+==================+=============+=======+=========+=========+
> | qwen3-4b | aime25    | mean_acc         | AIME2025-I  |   960 |  0.4594 | default |
> +----------+-----------+------------------+-------------+-------+---------+---------+
> | qwen3-4b | aime25    | mean_acc         | AIME2025-II |   960 |  0.4688 | default |
> +----------+-----------+------------------+-------------+-------+---------+---------+
> | qwen3-4b | aime25    | mean_acc         | OVERALL     |  1920 |  0.4641 | -       |
> +----------+-----------+------------------+-------------+-------+---------+---------+
> ```
> 
> 平均 acc 是 46.41，比你测的高一些。
> 
> 相关脚本和参数设置如下：
> 
> # vLLM 启动，2 x A100
> # vLLM API server version 0.10.1.1
> 
> CUDA_VISIBLE_DEVICES=5,6 TZ="Asia/Shanghai" \
>     python -m vllm.entrypoints.openai.api_server \
>     --model <model-path> \
>     --served-model-name qwen3-4b \
>     --tensor-parallel-size 2 \
>     --dtype bfloat16 \
>     --gpu-memory-utilization 0.97 \
>     --trust-remote-code \
>     --max-model-len 50000 \
>     --port 32104 \
>     --max-num-seqs 200 \
>     --enable-chunked-prefill --enable-prefix-caching
> # 评测和生成参数
> # evalscope 2.0.0
> ...
> 	eval_type="openai_api",
>     repeats=64,
> 	datasets=["aime25"], 
>     dataset_args={
>         "aime25": {  
>             "aggregation": "mean_and_pass_at_k",
>             "few_shot_num": 0
>         }
>     }, 
> 	eval_batch_size=100,
>     generation_config={
>         "max_tokens": 38912,
>         "temperature": 0.7,
>         "top_p": 0.8,
>         "top_k": 20,
>         "presence_penalty": 1.5
>     },  
>     timeout=60000,
>     stream=True, 
> ...
> 生成参数按 Qwen3 技术报告 P13 的描述：
> 
> > For Qwen3 models in the non-thinking mode, we configure the sampling hyperparameters with temperature = 0.7, top-p = 0.8, top-k = 20, and presence penalty = 1.5. For both the thinking and non-thinking modes, we set the max output length to 32,768 tokens, except AIME’24 and AIME’25 where we extend this length to 38,912 tokens to provide sufficient thinking space.

按照这里的参数设置，能够得到差不多的结果，我跑出来的结果为46.62。