Repository: modelscope/evalscope
Issue: R1类模型推理能力评测-推理问题
Root cause: Interface Contract Mismatch

vllm部署DeepSeek-R1-Distill-Qwen-32B
通过python代码评测模型：
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    model='Qwen2.5-7B-Instruct',   # 模型名称 (需要与部署时的模型名称一致)
    api_url='http://localhost:8000/v1',  # 推理服务地址
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,   # 评测类型，SERVICE表示评测推理服务
    datasets=[
        'data_collection',  # 数据集名称(固定为data_collection表示使用混合数据集)
    ],
    dataset_args={
        'data_collection': {
            'dataset_id': 'modelscope/R1-Distill-Math-Test'  # 数据集ID 或 数据集本地路径
        }
    },
    eval_batch_size=32,       # 发送请求的并发数
    generation_config={       # 模型推理配置
        'max_tokens': 20000,  # 最大生成token数，建议设置为较大值避免输出截断
        'temperature': 0.6,   # 采样温度 (deepseek 报告推荐值)
        'top_p': 0.95,        # top-p采样 (deepseek 报告推荐值)
        'n': 5                # 每个请求产生的回复数量 (注意 lmdeploy 目前只支持 n=1)
    },
    stream=True               # 是否使用流式请求，推荐设置为True防止请求超时
)

run_task(task_cfg=task_cfg)


日志：
2025-02-28 10:37:40,770 - evalscope - ERROR - Error when calling OpenAI API: 1 validation error for Choice
finish_reason
  Input should be 'stop', 'length', 'tool_calls', 'content_filter' or 'function_call' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
2025-02-28 10:37:55,635 - evalscope - ERROR - Error when calling OpenAI API: 1 validation error for Choice
finish_reason
  Input should be 'stop', 'length', 'tool_calls', 'content_filter' or 'function_call' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
2025-02-28 10:38:03,551 - evalscope - ERROR - Error when calling OpenAI API: 1 validation error for Choice
finish_reason
  Input should be 'stop', 'length', 'tool_calls', 'content_filter' or 'function_call' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error
2025-02-28 10:38:03,890 - evalscope - ERROR - Error when calling OpenAI API: 1 validation error for Choice
finish_reason
  Input should be 'stop', 'length', 'tool_calls', 'content_filter' or 'function_call' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error


请问这是什么问题


模型服务是用什么部署的

vllm部署DeepSeek-R1-Distill-Qwen-32B
________________________________
发件人: Yunlin Mao ***@***.***>
发送时间: 2025年2月28日 12:02 下午
收件人: modelscope/evalscope ***@***.***>
抄送: 丁冬仪(Danny) ***@***.***>; Author ***@***.***>
主题: Re: [modelscope/evalscope] R1类模型推理能力评测-推理问题 (Issue #345)


外部邮件/External Mail


模型服务是用什么部署的

―
Reply to this email directly, view it on GitHub<https://github.com/modelscope/evalscope/issues/345#issuecomment-2689663387>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPI2NA56WGHSMHPHIC42MU32R7NVNAVCNFSM6AAAAABYBLVV32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZGY3DGMZYG4>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

[Yunnglin]Yunnglin left a comment (modelscope/evalscope#345)<https://github.com/modelscope/evalscope/issues/345#issuecomment-2689663387>

模型服务是用什么部署的

―
Reply to this email directly, view it on GitHub<https://github.com/modelscope/evalscope/issues/345#issuecomment-2689663387>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPI2NA56WGHSMHPHIC42MU32R7NVNAVCNFSM6AAAAABYBLVV32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBZGY3DGMZYG4>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

________________________________
OPPO

本电子邮件及其附件含有OPPO公司的保密信息，仅限于邮件指明的收件人（包含个人及群组）使用。禁止任何人在未经授权的情况下以任何形式使用。如果您错收了本邮件，切勿传播、分发、复制、印刷或使用本邮件之任何部分或其所载之任何内容，并请立即以电子邮件通知发件人并删除本邮件及其附件。
网络通讯固有缺陷可能导致邮件被截留、修改、丢失、破坏或包含计算机病毒等不安全情况，OPPO对此类错误或遗漏而引致之任何损失概不承担责任并保留与本邮件相关之一切权利。
除非明确说明，本邮件及其附件无意作为在任何国家或地区之要约、招揽或承诺，亦无意作为任何交易或合同之正式确认。 发件人、其所属机构或所属机构之关联机构或任何上述机构之股东、董事、高级管理人员、员工或其他任何人（以下称“发件人”或“OPPO”）不因本邮件之误送而放弃其所享之任何权利，亦不对因故意或过失使用该等信息而引发或可能引发的损失承担任何责任。
文化差异披露：因全球文化差异影响，单纯以YES\OK或其他简单词汇的回复并不构成发件人对任何交易或合同之正式确认或接受，请与发件人再次确认以获得明确书面意见。发件人不对任何受文化差异影响而导致故意或错误使用该等信息所造成的任何直接或间接损害承担责任。
This e-mail and its attachments contain confidential information from OPPO, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you are not the intended recipient, please do not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
Electronic communications may contain computer viruses or other defects inherently, may not be accurately and/or timely transmitted to other systems, or may be intercepted, modified ,delayed, deleted or interfered. OPPO shall not be liable for any damages that arise or may arise from such matter and reserves all rights in connection with the email.
Unless expressly stated, this e-mail and its attachments are provided without any warranty, acceptance or promise of any kind in any country or region, nor constitute a formal confirmation or acceptance of any transaction or contract. The sender, together with its affiliates or any shareholder, director, officer, employee or any other person of any such institution (hereinafter referred to as "sender" or "OPPO") does not waive any rights and shall not be liable for any damages that arise or may arise from the intentional or negligent use of such information.
Cultural Differences Disclosure: Due to global cultural differences, any reply with only YES\OK or other simple words does not constitute any confirmation or acceptance of any transaction or contract, please confirm with the sender again to ensure clear opinion in written form. The sender shall not be responsible for any direct or indirect damages resulting from the intentional or misuse of such information.


看起来是因为finish reason有问题，有两个解决方法：
1. 不使用stream模式，`stream=False`
2. 安装main分支版本使用，应该修复了这个问题

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！