Repository: modelscope/evalscope
Issue: 代码生成类数据集进行评测的时候结果为0
Root cause: Documentation Deficiency

我先在本地部署的Qwen3-32B上对live_code_bench做测试，得分为0
日志如下：
`2025-05-12 13:47:17,089 - evalscope - INFO - Dump task config to ./outputs\20250512_134708\configs\task_config_5c0864.yaml
2025-05-12 13:47:17,092 - evalscope - INFO - {
    "model": "Qwen3-32B",
    "model_id": "Qwen3-32B",
    "model_args": {
        "revision": "master",
        "precision": "torch.float16"
    },
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "live_code_bench"
    ],
    "dataset_args": {
        "live_code_bench": {
            "name": "live_code_bench",
            "dataset_id": "AI-ModelScope/code_generation_lite",
            "model_adapter": "generation",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "release_latest"
            ],
            "metric_list": [
                "Pass@1"
            ],
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "### Question:\n{question_content}\n\n{format_prompt} ### Answer: (use the provided format with backticks)\n\n",
            "system_prompt": "You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.",
            "query_template": null,
            "pretty_name": "Live Code Bench",
            "filters": null,
            "extra_params": {
                "start_date": null,
                "end_date": null,
                "timeout": 6,
                "debug": false
            }
        }
    },
    "dataset_dir": "C:\\Users\\xrjxrj\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_length": 2048
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": null,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250512_134708",
    "outputs": null,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://192.168.2.120:10296/v1/chat/completions",
    "api_key": "xrj-qwen3",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
2025-05-12 13:47:17,095 - evalscope - INFO - **** Start evaluating on dataset AI-ModelScope/code_generation_lite ****
2025-05-12 13:47:17,096 - evalscope - INFO - Loading dataset from hub: AI-ModelScope/code_generation_lite
2025-05-12 13:47:18,154 - evalscope - INFO - Loading dataset: dataset_name: AI-ModelScope/code_generation_lite > subsets: ['release_latest']
2025-05-12 13:49:26,421 - evalscope - INFO - Use settings: > few_shot_num: 0, > few_shot_split: None, > target_eval_split: test
2025-05-14 12:52:55,296 - evalscope - INFO - Dump predictions to ./outputs\20250512_134708\predictions\Qwen3-32B\live_code_bench_release_latest.jsonl.
2025-05-14 12:58:55,563 - evalscope - INFO - Dump report: ./outputs\20250512_134708\reports\Qwen3-32B\live_code_bench.json 

2025-05-14 12:58:55,710 - evalscope - INFO - Report table: 
+-----------+-----------------+----------+----------------+-------+---------+---------+
| Model     | Dataset         | Metric   | Subset         |   Num |   Score | Cat.0   |
+===========+=================+==========+================+=======+=========+=========+
| Qwen3-32B | live_code_bench | Pass@1   | release_latest |  1055 |       0 | default |
+-----------+-----------------+----------+----------------+-------+---------+---------+ 

2025-05-14 12:58:55,710 - evalscope - INFO - **** Evaluation finished on AI-ModelScope/code_generation_lite ****
`
后来使用deepseek的api测了一下humaneval数据集，limit设了20，得分也是0。
所以说是模型预测的结果就是0,还是说评测过程有问题

平台：windows, python3.10.16, evalscope2.0.0

参考这里最佳实践：https://evalscope.readthedocs.io/zh-cn/latest/best_practice/eval_qwq.html#id5

设置`"filters": {"remove_until": "</think>"}`试试

此外，用`use_cache`参数可以复用模型生成结果直接跑评测

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！