Repository: modelscope/evalscope
Issue: hellaswag评测，直接调用模型和通过api调用，模型返回答案格式不一致
Root cause: Validation Gap

## 问题描述

hellaswag评测，直接调用模型和通过api调用，模型返回答案的格式并不是一致的。通过api调用的时候不是纯粹的答案A,B,C,D，导致匹配时评测为错误。为何有这样的差别呢？

打印的地方：
<img width="931" height="479" alt="Image" src="https://github.com/user-attachments/assets/27c969c7-0aa1-4660-bf84-fc407c10669b" />

两次调用的差别：

<img width="683" height="197" alt="Image" src="https://github.com/user-attachments/assets/ca12ab22-fd53-4f93-ac30-a27c7da3defa" />

<img width="823" height="346" alt="Image" src="https://github.com/user-attachments/assets/3291ae2e-5948-4b26-854a-748a68bb42b1" />

## EvalScope 版本（必填）
v0.17.0

## 执行的代码或指令

执行代码：
```
python -m evalscope.run --model Qwen/Qwen2.5-0.5B-Instruct --datasets hellaswag --limit 10
日志：
runpy.py:126: RuntimeWarning: 'evalscope.run' found in sys.modules after import of package 'evalscope', but prior to execution of 'evalscope.run'; this may result in unpredictable behaviour
2025-07-17 11:18:21,321 - evalscope - INFO - Args: Task config is provided with CommandLine type.
2025-07-17 11:18:24,312 - evalscope - INFO - Loading model Qwen/Qwen2.5-0.5B-Instruct ...
Downloading Model from https://www.modelscope.cn to directory: C:\Users\yerke\.cache\modelscope\hub\models\Qwen\Qwen2.5-0.5B-Instruct
Downloading Model from https://www.modelscope.cn to directory: C:\Users\yerke\.cache\modelscope\hub\models\Qwen\Qwen2.5-0.5B-Instruct
Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
2025-07-17 11:18:53,365 - evalscope - INFO - Dump task config to ./outputs\20250717_111821\configs\task_config_f2d422.yaml
2025-07-17 11:18:53,371 - evalscope - INFO - {
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "model_id": "Qwen2.5-0.5B-Instruct",
    "model_args": {
        "revision": "master",
        "precision": "torch.float16"
    },
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "hellaswag"
    ],
    "dataset_args": {
        "hellaswag": {
            "name": "hellaswag",
            "dataset_id": "modelscope/hellaswag",
            "model_adapter": "multiple_choice_logits",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "default"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": "train",
            "eval_split": "validation",
            "prompt_template": "{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "HellaSwag",
            "description": "HellaSwag is a benchmark for commonsense reasoning in natural language understanding tasks. It consists of multiple-choice questions where the model must select the most plausible continuation of a given context.",
            "tags": [
                "Commonsense",
                "MCQ",
                "Knowledge"
            ],
            "filters": null,
            "extra_params": {}
        }
    },
    "dataset_dir": "C:\\Users\\yerke\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_length": 2048,
        "max_new_tokens": 512,
        "do_sample": false,
        "top_k": 50,
        "top_p": 1.0,
        "temperature": 1.0
    },
    "eval_type": "checkpoint",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 10,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250717_111821",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-07-17 11:18:53,372 - evalscope - INFO - Start evaluating on dataset modelscope/hellaswag
2025-07-17 11:18:53,372 - evalscope - INFO - Loading dataset from hub: modelscope/hellaswag
2025-07-17 11:18:54,214 - evalscope - INFO - Loading dataset: dataset_name: modelscope/hellaswag > subsets: ['default']
2025-07-17 11:19:23,113 - evalscope - INFO - Use settings: > few_shot_num: 0, > few_shot_split: train, > target_eval_split: validation
Predicting(default): 100%|█████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  6.00it/s]
2025-07-17 11:19:26,822 - evalscope - INFO - Dump predictions to ./outputs\20250717_111821\predictions\Qwen2.5-0.5B-Instruct\hellaswag_default.jsonl.
2025-07-17 11:19:26,824 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,824 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,825 - evalscope - INFO - ===================result
 A
===================
Reviewing(default):   0%|                                                                       | 0/10 [00:00<?, ?it/s]2025-07-17 11:19:26,825 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,826 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,826 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,827 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,827 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,828 - evalscope - INFO - ===================result
 A
===================
2025-07-17 11:19:26,828 - evalscope - INFO - ===================result
 A
===================
Reviewing(default): 100%|████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 1665.46it/s]
2025-07-17 11:19:26,881 - evalscope - INFO -
modelscope/hellaswag report table:
+-----------------------+-----------+-----------------+----------+-------+---------+---------+
| Model                 | Dataset   | Metric          | Subset   |   Num |   Score | Cat.0   |
+=======================+===========+=================+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | hellaswag | AverageAccuracy | default  |    10 |     0.1 | default |
+-----------------------+-----------+-----------------+----------+-------+---------+---------+

2025-07-17 11:19:26,882 - evalscope - INFO - Skipping report analysis (`analysis_report=False`).
2025-07-17 11:19:26,885 - evalscope - INFO - Dump report to: ./outputs\20250717_111821\reports\Qwen2.5-0.5B-Instruct\hellaswag.json

2025-07-17 11:19:26,886 - evalscope - INFO - Evaluation finished on modelscope/hellaswag
2025-07-17 11:19:26,894 - evalscope - INFO - Overall report table:
+-----------------------+-----------+-----------------+----------+-------+---------+---------+
| Model                 | Dataset   | Metric          | Subset   |   Num |   Score | Cat.0   |
+=======================+===========+=================+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | hellaswag | AverageAccuracy | default  |    10 |     0.1 | default |
+-----------------------+-----------+-----------------+----------+-------+---------+---------+

2025-07-17 11:19:27,066 - evalscope - INFO - Finished evaluation for Qwen2.5-0.5B-Instruct on ['hellaswag']
2025-07-17 11:19:27,067 - evalscope - INFO - Output directory: ./outputs\20250717_111821

执行代码：
python -m evalscope.run --model Qwen2.5-0.5B-Instruct-Q4_K_S-gguf --api-url http://localhost:8080/v1/chat/completions --api-key empty --eval-type service --datasets hellaswag --limit 10
日志：
runpy.py:126: RuntimeWarning: 'evalscope.run' found in sys.modules after import of package 'evalscope', but prior to execution of 'evalscope.run'; this may result in unpredictable behaviour
2025-07-17 11:22:00,888 - evalscope - INFO - Args: Task config is provided with CommandLine type.
2025-07-17 11:22:03,783 - evalscope - INFO - Dump task config to ./outputs\20250717_112200\configs\task_config_c9f906.yaml
2025-07-17 11:22:03,798 - evalscope - INFO - {
    "model": "Qwen2.5-0.5B-Instruct-Q4_K_S-gguf",
    "model_id": "Qwen2.5-0.5B-Instruct-Q4_K_S-gguf",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "hellaswag"
    ],
    "dataset_args": {
        "hellaswag": {
            "name": "hellaswag",
            "dataset_id": "modelscope/hellaswag",
            "model_adapter": "multiple_choice_logits",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "default"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": "train",
            "eval_split": "validation",
            "prompt_template": "{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "HellaSwag",
            "description": "HellaSwag is a benchmark for commonsense reasoning in natural language understanding tasks. It consists of multiple-choice questions where the model must select the most plausible continuation of a given context.",
            "tags": [
                "Commonsense",
                "MCQ",
                "Knowledge"
            ],
            "filters": null,
            "extra_params": {}
        }
    },
    "dataset_dir": "C:\\Users\\yerke\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 2048,
        "temperature": 0.0
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 10,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250717_112200",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://localhost:8080/v1/chat/completions",
    "api_key": "empty",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-07-17 11:22:03,800 - evalscope - INFO - Start evaluating on dataset modelscope/hellaswag
2025-07-17 11:22:03,800 - evalscope - INFO - Loading dataset from hub: modelscope/hellaswag
2025-07-17 11:22:04,525 - evalscope - INFO - Loading dataset: dataset_name: modelscope/hellaswag > subsets: ['default']
2025-07-17 11:22:34,114 - evalscope - INFO - Use settings: > few_shot_num: 0, > few_shot_split: train, > target_eval_split: validation
Predicting(default): 100%|█████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.15it/s]
2025-07-17 11:22:41,018 - evalscope - INFO - Dump predictions to ./outputs\20250717_112200\predictions\Qwen2.5-0.5B-Instruct-Q4_K_S-gguf\hellaswag_default.jsonl.
2025-07-17 11:22:41,024 - evalscope - INFO - ===================result
 A. is using wrap to wrap a couple of skis.
===================
Reviewing(default):   0%|                                                                       | 0/10 [00:00<?, ?it/s]2025-07-17 11:22:41,025 - evalscope - INFO - ===================result
 A. swings and lands in her arms.
===================
2025-07-17 11:22:41,026 - evalscope - INFO - ===================result
 A. are then shown paddling along a river in a canoe while a woman talks.
===================
2025-07-17 11:22:41,026 - evalscope - INFO - ===================result
 A. runs into a car.
===================
2025-07-17 11:22:41,027 - evalscope - INFO - ===================result
 A. turns his body around on top of the red mat.
===================
2025-07-17 11:22:41,028 - evalscope - INFO - ===================result
 A. starts doing spins.
===================
2025-07-17 11:22:41,028 - evalscope - INFO - ===================result
 A. begins to play the harmonica

This ending makes the most sense because it captures the essence of the man playing the harmonica while looking at the camera, which aligns with the initial action described in the passage. The other options do not fit as well or make as much sense given the context and the actions described.
===================
2025-07-17 11:22:41,029 - evalscope - INFO - ===================result
 A. fight robots of evil and end with a to be continued.

This ending makes the most sense because it provides a clear conclusion to the cartoon animation video. It ends on a cliffhanger, leaving viewers wanting more information about what happens next in the story. The other options do not provide a logical or satisfying conclusion for the ongoing action sequence.
===================
2025-07-17 11:22:41,030 - evalscope - INFO - ===================result
 A.
===================
2025-07-17 11:22:41,030 - evalscope - INFO - ===================result
 B. The ending makes the most sense.

To determine which ending makes the most sense, let's analyze each option:

A. "cuts the center stone in half and blows it on to make it bigger."
- This ending suggests cutting a large stone into smaller pieces and then blowing them out. While this is an interesting idea, it doesn't directly address the process of breaking down the small stone into smaller pieces.

B. "grinds it hard to make the pieces smaller."
- Grinding a stone makes it smaller but does not involve breaking it down into smaller pieces. Instead, it involves using force to reduce its size.

C. "finally brings it back into view and adds them to the smaller ones to make a triangular shaped piece."
- This ending describes a process of bringing the small stone back into view and adding it to larger stones in a triangular shape. It aligns with breaking down the small stone, making it smaller, and then combining it with other smaller stones.

D. "starts to party with them and throws the pieces by hand while they celebrate."
- This ending describes throwing the small stone away and not involving any process of breaking it down or combining it with others. It doesn't make sense in the context of the problem.

Therefore, option C makes the most sense as it directly addresses the process of breaking a small stone into smaller pieces before combining them with other stones to form a triangular shape.
===================
Reviewing(default): 100%|████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 1250.80it/s]
2025-07-17 11:22:41,056 - evalscope - INFO -
modelscope/hellaswag report table:
+-----------------------------------+-----------+-----------------+----------+-------+---------+---------+
| Model                             | Dataset   | Metric          | Subset   |   Num |   Score | Cat.0   |
+===================================+===========+=================+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct-Q4_K_S-gguf | hellaswag | AverageAccuracy | default  |    10 |       0 | default |
+-----------------------------------+-----------+-----------------+----------+-------+---------+---------+

2025-07-17 11:22:41,056 - evalscope - INFO - Skipping report analysis (`analysis_report=False`).
2025-07-17 11:22:41,057 - evalscope - INFO - Dump report to: ./outputs\20250717_112200\reports\Qwen2.5-0.5B-Instruct-Q4_K_S-gguf\hellaswag.json

2025-07-17 11:22:41,057 - evalscope - INFO - Evaluation finished on modelscope/hellaswag
2025-07-17 11:22:41,068 - evalscope - INFO - Overall report table:
+-----------------------------------+-----------+-----------------+----------+-------+---------+---------+
| Model                             | Dataset   | Metric          | Subset   |   Num |   Score | Cat.0   |
+===================================+===========+=================+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct-Q4_K_S-gguf | hellaswag | AverageAccuracy | default  |    10 |       0 | default |
+-----------------------------------+-----------+-----------------+----------+-------+---------+---------+

2025-07-17 11:22:41,069 - evalscope - INFO - Finished evaluation for Qwen2.5-0.5B-Instruct-Q4_K_S-gguf on ['hellaswag']
2025-07-17 11:22:41,069 - evalscope - INFO - Output directory: ./outputs\20250717_112200
```
## 运行环境

- 操作系统：windows11
- Python版本：3.10.11


数据集race也是一样的情况

这两个数据集支持两种评测模式，默认是计算logits的MULTIPLE_CHOICE模式；但是通过api调用的模型时获取不到logits，会切换为GENERATION模式，从答案中匹配选项：
```python
model_adapter=OutputType.MULTIPLE_CHOICE,
output_types=[OutputType.MULTIPLE_CHOICE, OutputType.GENERATION],
```
答案处理上有点问题，我修复一下

> OutputType.GENERATION

那么这个我自己手动设置成OutputType.GENERATION，是否就正常了，也不会有其他影响

更新 v1.0.0 版本可以解决改问题

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！