Repository: modelscope/evalscope
Issue: KeyError: 'choices' when eval "ceval" with Qwen3-32B nothink mode
Root cause: Validation Gap

## 问题描述 / Issue Description

当测试Qwen3-32B的nothink模式时，如果直接使用 datasets 为ceval，会出现报错。查看相应的 ./outputs/20250507_142654/predictions/Qwen3-32B/ceval_computer_network.jsonl，发现部分问题完全没有回答，部分问题有正常回答。
另外，如果datasets设置为gsm8k，则完全正常。
使用 modelscope/EvalScope-Qwen3-Test ，测试1000条暂时也没有错误。

## EvalScope 版本 / Version (必须)
v0.15.1

## 使用的工具 / Tools Used
- [x] Native / 原生框架

## 执行的代码或指令 / Code or Commands Executed

如下python脚本，已经在localhost:17866 启动vllm，测试正常

```python
from evalscope import TaskConfig, run_task
import os

task_cfg = TaskConfig(
    model=os.getenv('MODEL', 'Qwen3-32B'),
    api_url=os.getenv('BASE_URL', 'http://127.0.0.1:17866/v1'),
    api_key=os.getenv('OAI_KEY'),
    eval_type='service',
    datasets=[
        os.getenv('TASKS','ceval'),
    ],
    dataset_args={
    },
    eval_batch_size=64,
    generation_config={
        'max_tokens': 20000,  # Max number of generated tokens, suggested to set a large value to avoid output truncation
        'temperature': 0.7,  # Sampling temperature (recommended value per Qwen report)
        'top_p': 0.8,  # top-p sampling (recommended value per Qwen report)
        'top_k': 20,  # top-k sampling (recommended value per Qwen report)
        'n': 1,  # Number of replies generated per request
        'chat_template_kwargs': {'enable_thinking': False}  # close thinking mode
    },
    timeout=600000,  # Timeout
    stream=True,  # Use streaming output
    limit=24,  # Set to 1000 samples for testing
)

run_task(task_cfg=task_cfg)
```

```bash
python script.py
```

## 错误日志 / Error Log

```
Predicting(computer_network): 100%|███████████████████████████████████████████████████| 19/19 [00:15<00:00,  1.22it/s]2025-05-07 14:32:33,334 - evalscope - INFO - Dump predictions to ./outputs/20250507_142654/predictions/Qwen3-32B/ceval_computer_network.jsonl.
Reviewing(computer_network):   0%|                                                             | 0/19 [00:00<?, ?it/s]
{'prompt_logprobs': None, 'model_spec': {'api_url': 'http://10.168.1.122:17866/v1', 'model_id': 'Qwen3-32B', 'api_key'
: 'ikang_2004'}, 'answer_id': 'answer-7ffb5613ba5d4ae972400ef073371edb', 'subset_name': 'computer_network', 'raw_input': {'id': 9, 'question': '关于传输层的面向连接服务的特性是____。', 'A': '既不保证可靠，也不保证按序交付', 'B': '不保证可靠，但保证按序交付', 'C': '保证可靠，但不保证按序交付', 'D': '既保证可靠，也保证按序交付', 'answer': 'D', 'explanation': ''}, 'index': 9}

Traceback (most recent call last):
  File "/data/deployer24/dl/src/evalscope/eval_qwen3_nothink.py", line 28, in <module>
    run_task(task_cfg=task_cfg)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    result = evaluate_model(task_cfg, outputs)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 126, in evaluate_model
    res_dict = evaluator.eval()
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 423, in eval
    reviews_list: list = self.get_reviews(subset_name=subset_name, answers_list=answers_list, **kwargs)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 302, in get_reviews
    review_d = future.result()
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 292, in process_single_review
    review_d = self._get_review(answer_d=answer_d, review_id=review_id, reviewer_spec=reviewer_spec)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 202, in _get_review
    choices = review_res[AnswerKeys.CHOICES]
KeyError: 'choices'
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [x] Ubuntu

- Python版本 / Python Version:
  - [x] 3.10



## 其他信息 / Additional Information

dump出来的 ./outputs/20250507_142654/predictions/Qwen3-32B/ceval_computer_network.jsonl 的前4行

```ceval_computer_network.jsonl
{"prompt_logprobs": null, "model_spec": {"api_url": "http://10.168.1.122:17866/v1", "model_id": "Qwen3-32B", "api_key": "ikang_2004"}, "answer_id": "answer-00a7a67dad757dde7bd80c40a4398f89", "subset_name": "computer_network", "raw_input": {"id": 10, "question": "____采用链路状态算法", "A": "RIP", "B": "OSPF", "C": "BGP-4", "D": "EGP", "answer": "B", "explanation": ""}, "index": 10}
{"prompt_logprobs": null, "model_spec": {"api_url": "http://10.168.1.122:17866/v1", "model_id": "Qwen3-32B", "api_key": "ikang_2004"}, "answer_id": "answer-9ac5bc54b70b57d130773e65d2f6bb29", "subset_name": "computer_network", "raw_input": {"id": 12, "question": "下列关于交换技术叙述错误的是____", "A": "电路交换在发送与接收方的物理链路上预留带宽", "B": "数据报交换可能出现分组乱序", "C": "数据报交换可能出现分组乱序", "D": "报文交换要求有较大缓存", "answer": "B", "explanation": ""}, "index": 12}
{"prompt_logprobs": null, "model_spec": {"api_url": "http://10.168.1.122:17866/v1", "model_id": "Qwen3-32B", "api_key": "ikang_2004"}, "answer_id": "answer-7ffb5613ba5d4ae972400ef073371edb", "subset_name": "computer_network", "raw_input": {"id": 9, "question": "关于传输层的面向连接服务的特性是____。", "A": "既不保证可靠，也不保证按序交付", "B": "不保证可靠，但保证按序交付", "C": "保证可靠，但不保证按序交付", "D": "既保证可靠，也保证按序交付", "answer": "D", "explanation": ""}, "index": 9}
{"id": "chatcmpl-0eeb206f417046e881f362d7872baa98", "choices": [{"finish_reason": "stop", "index": 0, "logprobs": null, "message": {"content": "答案是：C", "role": "assistant", "tool_calls": [], "reasoning_content": null}, "stop_reason": null}], "created": 1746599537, "model": "Qwen3-32B", "object": "chat.completion", "usage": {"completion_tokens": 5, "prompt_tokens": 104, "total_tokens": 109, "prompt_tokens_details": null}, "prompt_logprobs": null, "model_spec": {"api_url": "http://10.168.1.122:17866/v1", "model_id": "Qwen3-32B", "api_key": "ikang_2004"}, "answer_id": "answer-48124e02e12a03b1084cbb2f038d70b2", "subset_name": "computer_network", "raw_input": {"id": 2, "question": "TCP端对端通信作用>于____", "A": "主机之间", "B": "网络之间", "C": "进程之间", "D": "主机到网络", "answer": "C", "explanation": ""}, "index": 2}
...
```


看上去还是部分问题在对应的openai兼容服务中没返回导致的错误，只不过有时候2000多个问题只有两个报错，导致整个分数没法算很头疼。

勉强修改
evalscope/evaluator/evaluator.py 202行附近跳过这个错误
```
+        if AnswerKeys.CHOICES not in review_res:
+           print(review_res)
+           review_res[AnswerKeys.CHOICES] = []
        choices = review_res[AnswerKeys.CHOICES]
```


很奇怪为什么会没有返回结果，没有遇到过这种情况

> 很奇怪为什么会没有返回结果，没有遇到过这种情况

感觉这些错误总体还是和 [#492 ](https://github.com/modelscope/evalscope/issues/492)相关比较多。
有时候还会出现 “2025-05-07 17:11:40,250 - evalscope - ERROR - Error when calling emote API: 'MockValSer' object cannot be converted to 'SchemaSerializer'” 这类错误。

另外，在这里跳过之后，偶尔还会在下面这个地方碰到报错，我又是勉强跳过去了
```
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", lin
e 323, in compute_metrics
    try:
ValueError: max() arg is an empty sequence
```

就是     ```def compute_metrics(self, reviews_list: List[dict]) -> List[dict]:```
这个函数

修改前：
```
        review_res_list = []
        max_choices = max(
            len(review_d[AnswerKeys.CHOICES]) for review_d in reviews_list if review_d[ReviewKeys.REVIEWED])
```

修改后
```
        review_res_list = []
        choices_len = [len(review_d[AnswerKeys.CHOICES]) for review_d in reviews_list if review_d[ReviewKeys.REVIEWED]]
        if choices_len:
            max_choices = max(choices_len)
        else:
            max_choices = 1
            print(reviews_list)
```

完整错误log如下，看上去是结果都正常后算分出错。但是我看了predict下的jsonl看不到choices的数据。
```
Getting answers: 100%|██████████████████████████████████████████████████████████| 9169/9169 [5:26:09<00:00,  2.13s/it]
Getting reviews: 100%|████████████████████████████████████████████████████████████| 9169/9169 [04:20<00:00, 35.18it/s]
Getting scores:  12%|██████▉                                                  | 1121/9169 [00:00<00:00, 185776.40it/s]
Traceback (most recent call last):
  File "/data/deployer24/dl/src/evalscope/eval_qwen3_geneic_think.py", line 26, in <module>
    run_task(task_cfg=task_cfg)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    result = evaluate_model(task_cfg, outputs)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 126, in evaluate_model
    res_dict = evaluator.eval()
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 289, in eval
    scores = self.get_scores(reviews)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 281, in get_scores
    score = evaluator.get_score(review_d)
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/collections/evaluator.py", line 52, in get_score
    metric_score: List[dict] = self.compute_metrics(reviews_list=[review_d])
  File "/data/deployer24/miniforge3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 323, in compute_metrics
ValueError: max() arg is an empty sequence
```

Getting scores这一步能否单独执行来debug？

predict的文件压缩后仍然有200M.

可以用main分支代码再试试，修复了

另外一个极端情况，测试think mode时，会碰到返回不报错，choices=[]的结果，次数在另外一个文件中碰到错误。

导致这一点的原因是，远程是开了reason_parser的，思考过程在reason_content中，当思考token耗尽时如果还没有答案，那么返回的content就是""，从而choices=[]。

此时报错的文件是 evalscope/benchmarks/data_adapter.py 261行 compute_dict_metric 函数。

out of index错误，因为review_res_list是空的。

原始文件：
```
        if isinstance(review_res_list[0], list):
            review_res_list = [item for sublist in review_res_list for item in sublist]
```

```
        if len(review_res_list) > 0 and isinstance(review_res_list[0], list):
            review_res_list = [item for sublist in review_res_list for item in sublist]
```

> 另外一个极端情况，测试think mode时，会碰到返回不报错，choices=[]的结果，次数在另外一个文件中碰到错误。
> 
> 导致这一点的原因是，远程是开了reason_parser的，思考过程在reason_content中，当思考token耗尽时如果还没有答案，那么返回的content就是""，从而choices=[]。
> 
> 此时报错的文件是 evalscope/benchmarks/data_adapter.py 261行 compute_dict_metric 函数。
> 
> out of index错误，因为review_res_list是空的。
> 
> 原始文件：
> 
> ```
>         if isinstance(review_res_list[0], list):
>             review_res_list = [item for sublist in review_res_list for item in sublist]
> ```
> 
> ```
>         if len(review_res_list) > 0 and isinstance(review_res_list[0], list):
>             review_res_list = [item for sublist in review_res_list for item in sublist]
> ```

是的，这部分也在main分支更新了

> 是的，这部分也在main分支更新了

我拉取了最新的main，确实都能跑过了。

我还弄出命令行测试 qwen3 no_think的办法（因为 --generation-config 'max_tokens=20000,temperature=0.7,top_p=0.8,top_k=20,n=1,chat_template_kwargs={"enable_thinking": False}' 这种写法不行）

另外对sglang也兼容了。
我的语句如下
```
export BASE_URL=http://localhost:17966/v1
export MODEL=`curl ${BASE_URL}/models -H "Authorization: Bearer ${OAI_KEY}" | jq '.data[0] | .id // .root' | sed 's/"//g'`
echo $MODEL
export TASKS=ceval

evalscope eval \
 --model $MODEL \
 --api-url $BASE_URL \
 --api-key $OAI_KEY \
 --eval-type service \
 --datasets $TASKS \
 --stream --timeout 600000 --ignore-errors \
 --generation-config 'max_tokens=20000,temperature=0.7,top_p=0.8,top_k=20,n=1" \
 --eval-batch-size 64 \
 --dataset-args "$(echo '{"$TASKS": {"system_prompt": "You are a helpful assistant. /no_think."}}' | envsubst)"
```

唯一有点奇怪的是测试sglang的接口，就是会有固定的3个问题不返回任何结果，不知道有没有方便的办法debug具体的调用，如果能指定问题id debug也可以。这3个问题如果在open webui里面问，是可以得到正确答案的。

三个问题之一的报错
```
2025-05-09 18:18:23,954 - evalscope - ERROR - Failed to get answer for [{'data': ['以下是中国关于计算机网络考试的单项 选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：“答案是：LETTER”（不带引号），其中 LETTER 是 A、B、C、D 中的一个。\n问题：关于传输层的面向连接服务的特性是____。\nA. 既不保证可靠，也不保证按序交付\nB. 不保证可靠，但保证按序交付\nC. 保证可靠，但不保证按序交付\nD. 既保证可靠，也保证按序交付\n答案: '], 'index': 9, 'system_prompt': 'You are a helpful assistant. /no_think.', 'multi_choices': ['A', 'B', 'C', 'D'], 'raw_input': {'id': 9, 'question': '关于传输层的面向连接服务的特性是____。', 'A': '既不保证可靠，也不保证按序交付', 'B': '不保证可靠，但保证按序交付', 'C': '保证可靠，但不保证按序交付', 'D': '既保证可靠，也保证按序交付', 'answer': 'D', 'explanation': ''}}], due to 'MockValSer'
 object cannot be converted to 'SchemaSerializer'
2025-05-09 18:18:23,954 - evalscope - WARNING - `ignore_errors` is set to True. Dropping this prompt and continuing with evaluation.
```

1. `--generation-config`问题已修复，现在支持传json字符串格式，例如`--generation-config '{"do_sample":true,"temperature":0.5,"chat_template_kwargs":{"enable_thinking": false}}'`
2. `'MockValSer' object cannot be converted to 'SchemaSerializer'` 报错问题应该是OpenAI API的问题，参考 https://github.com/pydantic/pydantic/issues/7713


> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！