Repository: modelscope/evalscope
Issue: '<' not supported between instances of 'str' and 'int'
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

请简要描述您遇到的问题。

## EvalScope 版本（必填）
evalscope 2.0.0

## 执行的代码或指令
task_cfg = {
            'api_url': server_url,
            'model': self.model_name,
            'datasets': ['gsm8k', 'aime24', 'aime25', 'math_500'],
            'work_dir': self.output_dir,
            'eval_type': 'openai_api',
            'eval_batch_size': '12',
            'generation_config': {
                "do_sample":True,
                "temperature":0.7,
                "top_p":0.8,
                "top_k":20,
                "max_tokens":20480,
                "frequency_penalty":1.5,
                "chat_template_kwargs":{
                    "enable_thinking": False
                }
            },
            'dataset_args': {
                "system_prompt": system_prompt
            },
            'use_cache': cache_dir,
            'ignore_errors': True
        }
        run_task(task_cfg=task_cfg)

请提供您执行的主要代码或指令。

## 错误日志

2025-10-20 16:52:27 - evalscope - INFO: Dump task config to results-v2/20251020_165227/configs/task_config.yaml
2025-10-20 16:52:27 - evalscope - INFO: {
    "model": "qwen3_14b_base",
    "model_id": "qwen3_14b_base",
    "model_args": {},
    "model_task": "text_generation",
    "chat_template": null,
    "datasets": [
        "gsm8k",
        "aime24",
        "aime25",
        "math_500",
        "amc"
    ],
    "dataset_args": {
        "system_prompt": "Your role as an assistant involves thoroughly exploring questions through a systematic long thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution. In the Thought section, detail your reasoning process using the specified format: 
 {thought with steps separated with '\n\n'} 
 Each step should include detailed considerations such as analisying questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The solution should remain a logical, accurate, concise expression style and detail necessary step needed to reach the conclusion, formatted as follows: 
 {final formatted, precise, and clear solution} 
 Now, try to solve the following question through the above guidelines:",
        "gsm8k": {
            "name": "gsm8k",
            "dataset_id": "AI-ModelScope/gsm8k",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "main"
            ],
            "default_subset": "default",
            "few_shot_num": 4,
            "few_shot_random": false,
            "train_split": "train",
            "eval_split": "test",
            "prompt_template": "Solve the following math problem step by step. The last line of your response should be of the form \"ANSWER: $ANSWER\" (without quotes) where $ANSWER is the answer to the problem.\n\n{question}\n\nRemember to put your answer on its own line at the end in the form \"ANSWER: $ANSWER\" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.\n\nReasoning:\n",
            "few_shot_prompt_template": "Here are some examples of how to solve similar problems:\n\n{fewshot}\n\nSolve the following math problem step by step. The last line of your response should be of the form \"ANSWER: $ANSWER\" (without quotes) where $ANSWER is the answer to the problem.\n\n{question}\n\nRemember to put your answer on its own line at the end in the form \"ANSWER: $ANSWER\" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.\n\nReasoning:\n",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "GSM8K",
            "description": "GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.",
            "tags": [
                "Math",
                "Reasoning"
            ],
            "filters": null,
            "metric_list": [
                "acc"
            ],
            "aggregation": "mean",
            "shuffle": false,
            "shuffle_choices": false,
            "review_timeout": null,
            "extra_params": {}
        },
        "aime24": {
            "name": "aime24",
            "dataset_id": "HuggingFaceH4/aime_2024",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "default"
            ],
            "default_subset": "default",
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "train",
            "prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
            "few_shot_prompt_template": null,
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "AIME-2024",
            "description": "The AIME 2024 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
            "tags": [
                "Math",
                "Reasoning"
            ],
            "filters": null,
            "metric_list": [
                {
                    "acc": {
                        "numeric": true
                    }
                }
            ],
            "aggregation": "mean",
            "shuffle": false,
            "shuffle_choices": false,
            "review_timeout": null,
            "extra_params": {}
        },
        "aime25": {
            "name": "aime25",
            "dataset_id": "opencompass/AIME2025",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "AIME2025-I",
                "AIME2025-II"
            ],
            "default_subset": "default",
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
            "few_shot_prompt_template": null,
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "AIME-2025",
            "description": "The AIME 2025 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
            "tags": [
                "Math",
                "Reasoning"
            ],
            "filters": null,
            "metric_list": [
                {
                    "acc": {
                        "numeric": true
                    }
                }
            ],
            "aggregation": "mean",
            "shuffle": false,
            "shuffle_choices": false,
            "review_timeout": null,
            "extra_params": {}
        },
        "math_500": {
            "name": "math_500",
            "dataset_id": "AI-ModelScope/MATH-500",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "Level 1",
                "Level 2",
                "Level 3",
                "Level 4",
                "Level 5"
            ],
            "default_subset": "default",
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
            "few_shot_prompt_template": null,
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "MATH-500",
            "description": "MATH-500 is a benchmark for evaluating mathematical reasoning capabilities of AI models. It consists of 500 diverse math problems across five levels of difficulty, designed to test a model's ability to solve complex mathematical problems by generating step-by-step solutions and providing the correct final answer.",
            "tags": [
                "Math",
                "Reasoning"
            ],
            "filters": null,
            "metric_list": [
                {
                    "acc": {
                        "numeric": true
                    }
                }
            ],
            "aggregation": "mean",
            "shuffle": false,
            "shuffle_choices": false,
            "review_timeout": null,
            "extra_params": {}
        },
        "amc": {
            "name": "amc",
            "dataset_id": "evalscope/amc_22-24",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "amc22",
                "amc23",
                "amc24"
            ],
            "default_subset": "default",
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": null,
            "prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
            "few_shot_prompt_template": null,
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "AMC",
            "description": "AMC (American Mathematics Competitions) is a series of mathematics competitions for high school students.",
            "tags": [
                "Math",
                "Reasoning"
            ],
            "filters": null,
            "metric_list": [
                {
                    "acc": {
                        "numeric": true
                    }
                }
            ],
            "aggregation": "mean",
            "shuffle": false,
            "shuffle_choices": false,
            "review_timeout": null,
            "extra_params": {}
        }
    },
    "dataset_dir": "/mnt/workspace/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "repeats": 1,
    "generation_config": {
        "batch_size": "12",
        "max_tokens": 20480,
        "top_p": 0.8,
        "temperature": 0.7,
        "frequency_penalty": 1.5,
        "do_sample": true,
        "top_k": 20,
        "chat_template_kwargs": {
            "enable_thinking": false
        }
    },
    "eval_type": "openai_api",
    "eval_backend": "Native",
    "eval_config": null,
    "limit": null,
    "eval_batch_size": "12",
    "use_cache": null,
    "rerun_review": false,
    "work_dir": "results-v2/20251020_165227",
    "ignore_errors": true,
    "debug": false,
    "seed": 42,
    "api_url": "http://0.0.0.0:8000/v1",
    "timeout": null,
    "stream": null,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false,
    "use_sandbox": false,
    "sandbox_type": "docker",
    "sandbox_manager_config": {},
    "sandbox_config": {}
}
[2025-10-20 16:52:27] INFO:     127.0.0.1:57538 - "POST /generate HTTP/1.1" 200 OK
[2025-10-20 16:52:27] The server is fired up and ready to roll!
Processing records: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:00<00:00, 270408.47it/s]
Processing records: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 90687.65it/s]
Evaluation failed: '<' not supported between instances of 'str' and 'int'




控制台中有其他报错信息吗，从日志中看不出具体的原因

> 控制台中有其他报错信息吗，从日志中看不出具体的原因

没其他的日志输出来，run_task(task_cfg=task_cfg)这里抛出的