Repository: modelscope/evalscope
Issue: 原生框架评测humaneval分数为0
Root cause: Environment Incompatibility

## 问题描述

humaneval评测集无论是调few_shot_num，Pass@3都是0分，今天拿0.17.2的试一下还是0分，哪里的参数需要调整的吗，有能得到有效分数的最佳指令吗

## EvalScope 版本（必填）
v0.17.2

## 执行的代码或指令

evalscope eval --model D:\models\Qwen2.5-Coder-7B-Instruct --model-args "device_map=cuda" --model-id Qwen/Qwen2.5-Coder-7B-Instruct  --datasets humaneval --generation-config n=3,max_tokens=25000 --dataset-args "{\"humaneval\": {\"metric_list\": [\"Pass@3\"]}}" --limit 50

## 错误日志

evalscope eval --model D:\models\Qwen2.5-Coder-7B-Instruct --model-args "device_map=cuda" --model-id Qwen/Qwen2.5-Coder-7B-Instruct  --datasets humaneval --generation-config n=3,max_tokens=25000 --dataset-args "{\"humaneval\": {\"metric_list\": [\"Pass@3\"]}}" --limit 50
2025-07-23 10:55:28,332 - evalscope - INFO - Args: Task config is provided with CommandLine type.
2025-07-23 10:55:31,900 - evalscope - INFO - Loading model D:\models\Qwen2.5-Coder-7B-Instruct ...
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████| 4/4 [00:17<00:00,  4.39s/it]
2025-07-23 10:55:55,018 - evalscope - WARNING - Got local model dir: D:\models\Qwen2.5-Coder-7B-Instruct
2025-07-23 10:55:55,018 - evalscope - INFO - Updating generation config ...
2025-07-23 10:55:55,640 - evalscope - INFO - Dump task config to ./outputs\20250723_105528\configs\task_config_4fbc8e.yaml
2025-07-23 10:55:55,643 - evalscope - INFO - {
    "model": "D:\\models\\Qwen2.5-Coder-7B-Instruct",
    "model_id": "Qwen/Qwen2.5-Coder-7B-Instruct",
    "model_args": {
        "device_map": "cuda"
    },
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "humaneval"
    ],
    "dataset_args": {
        "humaneval": {
            "name": "humaneval",
            "dataset_id": "modelscope/humaneval",
            "model_adapter": "generation",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "openai_humaneval"
            ],
            "metric_list": [
                "Pass@3"
            ],
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "HumanEval",
            "description": "HumanEval is a benchmark for evaluating the ability of code generation models to write Python functions based on given specifications. It consists of programming tasks with a defined input-output behavior.",
            "tags": [
                "Coding"
            ],
            "filters": null,
            "extra_params": {
                "num_workers": 4,
                "timeout": 4
            }
        }
    },
    "dataset_dir": "C:\\Users\\yerke\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "n": 3,
        "max_tokens": 25000
    },
    "eval_type": "checkpoint",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 50,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250723_105528",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-07-23 10:55:55,648 - evalscope - INFO - Start evaluating on dataset modelscope/humaneval
2025-07-23 10:55:55,648 - evalscope - INFO - Loading dataset from hub: modelscope/humaneval
2025-07-23 10:55:56,021 - evalscope - INFO - Loading dataset: dataset_name: modelscope/humaneval > subsets: ['openai_humaneval']
2025-07-23 10:56:03,552 - evalscope - INFO - Use settings: > few_shot_num: 0, > few_shot_split: None, > target_eval_split: test
Predicting(openai_humaneval): 100%|██████████████████████████████████████████████████████████████████████████████| 50/50 [08:16<00:00,  9.93s/it]
2025-07-23 11:04:20,008 - evalscope - INFO - Dump predictions to ./outputs\20250723_105528\predictions\Qwen/Qwen2.5-Coder-7B-Instruct\humaneval_openai_humaneval.jsonl.
Reviewing(openai_humaneval): 100%|███████████████████████████████████████████████████████████████████████████████| 50/50 [00:07<00:00,  6.35it/s]
2025-07-23 11:04:27,922 - evalscope - INFO -
modelscope/humaneval report table:
+--------------------------------+-----------+----------+------------------+-------+---------+---------+
| Model                          | Dataset   | Metric   | Subset           |   Num |   Score | Cat.0   |
+================================+===========+==========+==================+=======+=========+=========+
| Qwen/Qwen2.5-Coder-7B-Instruct | humaneval | Pass@3   | openai_humaneval |    50 |       0 | default |
+--------------------------------+-----------+----------+------------------+-------+---------+---------+

2025-07-23 11:04:27,923 - evalscope - INFO - Skipping report analysis (`analysis_report=False`).
2025-07-23 11:04:27,927 - evalscope - INFO - Dump report to: ./outputs\20250723_105528\reports\Qwen/Qwen2.5-Coder-7B-Instruct\humaneval.json

2025-07-23 11:04:27,928 - evalscope - INFO - Evaluation finished on modelscope/humaneval
2025-07-23 11:04:27,964 - evalscope - INFO - Overall report table:
+--------------------------------+-----------+----------+------------------+-------+---------+---------+
| Model                          | Dataset   | Metric   | Subset           |   Num |   Score | Cat.0   |
+================================+===========+==========+==================+=======+=========+=========+
| Qwen/Qwen2.5-Coder-7B-Instruct | humaneval | Pass@3   | openai_humaneval |    50 |       0 | default |
+--------------------------------+-----------+----------+------------------+-------+---------+---------+

2025-07-23 11:04:28,138 - evalscope - INFO - Finished evaluation for Qwen/Qwen2.5-Coder-7B-Instruct on ['humaneval']
2025-07-23 11:04:28,138 - evalscope - INFO - Output directory: ./outputs\20250723_105528

## 运行环境

- 操作系统：windows11
- Python版本：3.10.11

## 其他信息

如果有其他相关信息，请在此处提供。


运行evalscope app命令，截图看一下模型的输出样例

> 运行evalscope app命令，截图看一下模型的输出样例

<img width="1342" height="816" alt="Image" src="https://github.com/user-attachments/assets/a037a1a1-fbf4-4121-9a42-7383fedbc674" />

数据详情报错了，加载不出来，我姑且在reviews复制了一道题：

{
    "model": "D:\models\Qwen2.5-Coder-7B-Instruct",
    "object": "chat.completion",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "```python
from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    for i in range(len(numbers)):
        for j in range(i + 1, len(numbers)):
            if abs(numbers[i] - numbers[j]) < threshold:
                return True
    return False
```"
            },
            "finish_reason": "stop",
            "review": {
                "gold": "*Same as Input*",
                "pred": "from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    for i in range(len(numbers)):
        for j in range(i + 1, len(numbers)):
            if abs(numbers[i] - numbers[j]) < threshold:
                return True
    return False
",
                "result": 0
            }
        }
    ],
    "created": 1753241576,
    "usage": {
        "prompt_tokens": 174,
        "completion_tokens": 164,
        "total_tokens": 338
    },
    "model_spec": {
        "model_id": "D:\models\Qwen2.5-Coder-7B-Instruct",
        "device_map": "cuda",
        "torch_dtype": "auto"
    },
    "answer_id": "answer-4c2c6cea0c64e717ed91a6de5465b6e0",
    "subset_name": "openai_humaneval",
    "raw_input": {
        "task_id": "HumanEval/0",
        "prompt": "from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
",
        "canonical_solution": "    for idx, elem in enumerate(numbers):
        for idx2, elem2 in enumerate(numbers):
            if idx != idx2:
                distance = abs(elem - elem2)
                if distance < threshold:
                    return True

    return False
",
        "test": "

METADATA = {
    'author': 'jt',
    'dataset': 'test'
}


def check(candidate):
    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True
    assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False
    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True
    assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False
    assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True
    assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True
    assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False

",
        "entry_point": "has_close_elements"
    },
    "index": 0,
    "reviewed": true,
    "review_id": "review-d49e6c7e2954f07309a5de717f193b10",
    "reviewer_spec": {
        "metric": [
            "Pass@3"
        ],
        "reviewer": [
            "Evaluator"
        ],
        "revision": [
            "default"
        ]
    },
    "review_time": 1753242054.002642
}

模型这个结果是正确的，有点奇怪，这个目录下的内容打个zip包传上来看看 ./outputs\20250723_105528

> 模型这个结果是正确的，有点奇怪，这个目录下的内容打个zip包传上来看看 ./outputs\20250723_105528

好的
[20250723_105528.zip](https://github.com/user-attachments/files/21379536/20250723_105528.zip)

evalscope app加载报错是因为，model id `Qwen/Qwen2.5-Coder-7B-Instruct`中出现了`/`，被解析成了路径，导致加载失败。把model ID改为 `Qwen2.5-Coder-7B-Instruct`再评测试试

> evalscope app加载报错是因为，model id `Qwen/Qwen2.5-Coder-7B-Instruct`中出现了`/`，被解析成了路径，导致加载失败。把model ID改为 `Qwen2.5-Coder-7B-Instruct`再评测试试

测了，evalscope app正常了

evalscope eval --model D:\models\Qwen2.5-Coder-7B-Instruct --model-args "device_map=cuda" --model-id Qwen2.5-Coder-7B-Instruct --datasets humaneval --generation-config n=3,max_tokens=25000  --dataset-args "{\"humaneval\": {\"metric_list\": [\"Pass@3\"]}}" --limit 50
2025-07-23 18:19:46,553 - evalscope - INFO - Args: Task config is provided with CommandLine type.
2025-07-23 18:20:09,693 - evalscope - INFO - Loading model D:\models\Qwen2.5-Coder-7B-Instruct ...
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 4/4 [00:56<00:00, 14.22s/it]
2025-07-23 18:21:15,375 - evalscope - WARNING - Got local model dir: D:\models\Qwen2.5-Coder-7B-Instruct
2025-07-23 18:21:15,384 - evalscope - INFO - Updating generation config ...
2025-07-23 18:21:16,126 - evalscope - INFO - Dump task config to ./outputs\20250723_181946\configs\task_config_256a6e.yaml
2025-07-23 18:21:16,138 - evalscope - INFO - {
    "model": "D:\\models\\Qwen2.5-Coder-7B-Instruct",
    "model_id": "Qwen2.5-Coder-7B-Instruct",
    "model_args": {
        "device_map": "cuda"
    },
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "humaneval"
    ],
    "dataset_args": {
        "humaneval": {
            "name": "humaneval",
            "dataset_id": "modelscope/humaneval",
            "model_adapter": "generation",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "openai_humaneval"
            ],
            "metric_list": [
                "Pass@3"
            ],
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "HumanEval",
            "description": "HumanEval is a benchmark for evaluating the ability of code generation models to write Python functions based on given specifications. It consists of programming tasks with a defined input-output behavior.",
            "tags": [
                "Coding"
            ],
            "filters": null,
            "extra_params": {
                "num_workers": 4,
                "timeout": 4
            }
        }
    },
    "dataset_dir": "C:\\Users\\yerke\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "n": 3,
        "max_tokens": 25000
    },
    "eval_type": "checkpoint",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 50,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250723_181946",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-07-23 18:21:16,140 - evalscope - INFO - Start evaluating on dataset modelscope/humaneval
2025-07-23 18:21:16,140 - evalscope - INFO - Loading dataset from hub: modelscope/humaneval
2025-07-23 18:21:17,633 - evalscope - INFO - Loading dataset: dataset_name: modelscope/humaneval > subsets: ['openai_humaneval']
2025-07-23 18:21:34,814 - evalscope - INFO - Use settings: > few_shot_num: 0, > few_shot_split: None, > target_eval_split: test
Predicting(openai_humaneval): 100%|████████████████████████████████████████████████████| 50/50 [06:43<00:00,  8.08s/it]
2025-07-23 18:28:18,739 - evalscope - INFO - Dump predictions to ./outputs\20250723_181946\predictions\Qwen2.5-Coder-7B-Instruct\humaneval_openai_humaneval.jsonl.
Reviewing(openai_humaneval): 100%|█████████████████████████████████████████████████████| 50/50 [00:07<00:00,  6.42it/s]
2025-07-23 18:28:26,593 - evalscope - INFO -
modelscope/humaneval report table:
+---------------------------+-----------+----------+------------------+-------+---------+---------+
| Model                     | Dataset   | Metric   | Subset           |   Num |   Score | Cat.0   |
+===========================+===========+==========+==================+=======+=========+=========+
| Qwen2.5-Coder-7B-Instruct | humaneval | Pass@3   | openai_humaneval |    50 |       0 | default |
+---------------------------+-----------+----------+------------------+-------+---------+---------+

2025-07-23 18:28:26,593 - evalscope - INFO - Skipping report analysis (`analysis_report=False`).
2025-07-23 18:28:26,594 - evalscope - INFO - Dump report to: ./outputs\20250723_181946\reports\Qwen2.5-Coder-7B-Instruct\humaneval.json

2025-07-23 18:28:26,594 - evalscope - INFO - Evaluation finished on modelscope/humaneval
2025-07-23 18:28:26,606 - evalscope - INFO - Overall report table:
+---------------------------+-----------+----------+------------------+-------+---------+---------+
| Model                     | Dataset   | Metric   | Subset           |   Num |   Score | Cat.0   |
+===========================+===========+==========+==================+=======+=========+=========+
| Qwen2.5-Coder-7B-Instruct | humaneval | Pass@3   | openai_humaneval |    50 |       0 | default |
+---------------------------+-----------+----------+------------------+-------+---------+---------+

2025-07-23 18:28:26,793 - evalscope - INFO - Finished evaluation for Qwen2.5-Coder-7B-Instruct on ['humaneval']
2025-07-23 18:28:26,794 - evalscope - INFO - Output directory: ./outputs\20250723_181946

<img width="1176" height="815" alt="Image" src="https://github.com/user-attachments/assets/43b1f73b-3206-4dec-b55d-4046af40e7e0" />

<img width="1255" height="904" alt="Image" src="https://github.com/user-attachments/assets/90728c1a-9ccc-416b-8fbb-94396446ca6d" />

<img width="1272" height="797" alt="Image" src="https://github.com/user-attachments/assets/506c0b98-d650-45ae-91da-bf36b19ec89b" />

[20250723_181946.zip](https://github.com/user-attachments/files/21385981/20250723_181946.zip)

我拿你的模型输出评测，结果是正常的：

<img width="1700" height="180" alt="Image" src="https://github.com/user-attachments/assets/d56d745c-d1fc-4e97-8349-747947fab95e" />

有可能跟windows系统有关，尝试使用WSL测试

我用wsl测试了，确实是拿到有效分数了，这是什么原因导致的呢？有办法能让windows也正常得分吗

> 我用wsl测试了，确实是拿到有效分数了，这是什么原因导致的呢？有办法能让windows也正常得分吗

打印后看到报错了，似乎是说Windows不支持 signal.setitimer？
<img width="716" height="185" alt="Image" src="https://github.com/user-attachments/assets/82a7d694-3aaf-4530-9119-c20e026d65b7" />

<img width="732" height="177" alt="Image" src="https://github.com/user-attachments/assets/b47459c3-b7cd-4c88-ae79-e9ae2b93f7b1" />

这个暂时不好解决，match中使用了 humaneval的官方代码，里面的signal是windows不支持的。后续会上线sandbox功能，在独立可复现的环境中执行代码。

> 这个暂时不好解决，match中使用了 humaneval的官方代码，里面的signal是windows不支持的。后续会上线sandbox功能，在独立可复现的环境中执行代码。

好的好的👍