Repository: modelscope/evalscope
Issue: return _winapi.DuplicateHandle( OSError: [WinError 6] 句柄无效。
Root cause: Environment Incompatibility

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\anaconda3\envs\evalscope\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\ProgramData\anaconda3\envs\evalscope\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] 句柄无效。
Getting reviews:  69%|██████▉   | 605/880 [01:40<01:44,  2.64it/s]2025-05-24 17:35:58,787 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
```

## EvalScope 版本（必填）
v0.16.0

## 使用的工具
- [x ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

```python
from evalscope import TaskConfig, run_task
dataset_path=r'C:\Users\Administrator\PycharmProjects\llmEval\EvalScope\dataset\weighted_mixed_data.jsonl'
task_cfg = TaskConfig(
    model='Qwen3-14B-AWQ',
    api_url='http://127.0.0.1:8889/v1/chat/completions',
    eval_type='service',
    datasets=[
        'data_collection',
    ],
    dataset_args={
        'data_collection': {
            'dataset_id': dataset_path,
            'filters': {'remove_until': '</think>'}  # 过滤掉思考的内容
        }
    },
    eval_batch_size=15,
    generation_config={
        'max_tokens': 30000,  # 最大生成token数，建议设置为较大值避免输出截断
        'temperature': 0.6,  # 采样温度 (qwen 报告推荐值)
        'top_p': 0.95,  # top-p采样 (qwen 报告推荐值)
        'top_k': 20,  # top-k采样 (qwen 报告推荐值)
        'n': 1,  # 每个请求产生的回复数量
    },
    timeout=60000,  # 超时时间
    stream=True,  # 是否使用流式输出
    # limit=2000,  # 设置为100条数据进行测试
    use_cache=r"./outputs/20250524_152506"
)

run_task(task_cfg=task_cfg)
```

## 错误日志

```bash
C:\ProgramData\anaconda3\envs\evalscope\python.exe C:\Users\Administrator\PycharmProjects\llmEval\EvalScope\H800\autodl_h800.py 
2025-05-24 17:34:00,366 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
2025-05-24 17:34:00,381 - evalscope - INFO - Set resume from ./outputs/20250524_152506
2025-05-24 17:34:04,916 - evalscope - INFO - Loading dataset from C:\Users\Administrator\PycharmProjects\llmEval\EvalScope\dataset\weighted_mixed_data.jsonl
2025-05-24 17:34:10,057 - evalscope - WARNING - Output type is set to logits. This is not supported for service evaluation. Setting output type to generation by default.
2025-05-24 17:34:15,274 - evalscope - INFO - Dump task config to ./outputs/20250524_152506\configs\task_config_7fd2da.yaml
2025-05-24 17:34:15,276 - evalscope - INFO - {
    "model": "Qwen3-14B-AWQ",
    "model_id": "Qwen3-14B-AWQ",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "data_collection"
    ],
    "dataset_args": {
        "data_collection": {
            "dataset_id": "C:\\Users\\Administrator\\PycharmProjects\\llmEval\\EvalScope\\dataset\\weighted_mixed_data.jsonl",
            "filters": {
                "remove_until": "</think>"
            }
        }
    },
    "dataset_dir": "C:\\Users\\Administrator\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 30000,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20,
        "n": 1
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": null,
    "eval_batch_size": 15,
    "mem_cache": false,
    "use_cache": "./outputs/20250524_152506",
    "work_dir": "./outputs/20250524_152506",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8889/v1/chat/completions",
    "api_key": "EMPTY",
    "timeout": 60000,
    "stream": true,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
2025-05-24 17:34:16,135 - evalscope - INFO - Reuse from ./outputs/20250524_152506\predictions\Qwen3-14B-AWQ\weighted_mixed_data.jsonl. Loaded 880 samples, remain 0 samples.
Getting answers: 0it [00:00, ?it/s]
2025-05-24 17:34:16,136 - evalscope - INFO - use_cache=./outputs/20250524_152506, reloading the review file: ./outputs/20250524_152506\reviews\Qwen3-14B-AWQ
Getting reviews:   0%|          | 0/880 [00:00<?, ?it/s]2025-05-24 17:34:18,363 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
2025-05-24 17:34:18,376 - evalscope - INFO - Set resume from ./outputs/20250524_152506
2025-05-24 17:34:23,077 - evalscope - INFO - Loading dataset from C:\Users\Administrator\PycharmProjects\llmEval\EvalScope\dataset\weighted_mixed_data.jsonl
2025-05-24 17:34:28,213 - evalscope - WARNING - Output type is set to logits. This is not supported for service evaluation. Setting output type to generation by default.
2025-05-24 17:34:33,327 - evalscope - INFO - Dump task config to ./outputs/20250524_152506\configs\task_config_7fd2da.yaml
2025-05-24 17:34:33,330 - evalscope - INFO - {
    "model": "Qwen3-14B-AWQ",
    "model_id": "Qwen3-14B-AWQ",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "data_collection"
    ],
    "dataset_args": {
        "data_collection": {
            "dataset_id": "C:\\Users\\Administrator\\PycharmProjects\\llmEval\\EvalScope\\dataset\\weighted_mixed_data.jsonl",
            "filters": {
                "remove_until": "</think>"
            }
        }
    },
    "dataset_dir": "C:\\Users\\Administrator\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 30000,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20,
        "n": 1
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": null,
    "eval_batch_size": 15,
    "mem_cache": false,
    "use_cache": "./outputs/20250524_152506",
    "work_dir": "./outputs/20250524_152506",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8889/v1/chat/completions",
    "api_key": "EMPTY",
    "timeout": 60000,
    "stream": true,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
2025-05-24 17:34:34,311 - evalscope - INFO - Reuse from ./outputs/20250524_152506\predictions\Qwen3-14B-AWQ\weighted_mixed_data.jsonl. Loaded 880 samples, remain 0 samples.
Getting answers: 0it [00:00, ?it/s]
2025-05-24 17:34:34,313 - evalscope - INFO - use_cache=./outputs/20250524_152506, reloading the review file: ./outputs/20250524_152506\reviews\Qwen3-14B-AWQ
Getting reviews: 100%|██████████| 880/880 [00:01<00:00, 578.38it/s]
Getting scores: 100%|██████████| 880/880 [00:00<00:00, 381103.51it/s]
2025-05-24 17:34:35,912 - evalscope - INFO - subset_level Report:
+-----------+-----------------+-----------------+------------------------------------------+---------------+-------+
| task_type |     metric      |  dataset_name   |               subset_name                | average_score | count |
+-----------+-----------------+-----------------+------------------------------------------+---------------+-------+
|   code    |     Pass@1      | live_code_bench |                  v5_v6                   |      0.0      |  90   |
|   math    | AverageAccuracy |      gsm8k      |                   main                   |    0.9444     |  90   |
| reasoning | AverageAccuracy |    hellaswag    |                 default                  |    0.7889     |  90   |
| knowledge |  AveragePass@1  |      gpqa       |               gpqa_diamond               |    0.6333     |  90   |
|   exam    | AverageAccuracy |      iquiz      |                    EQ                    |    0.6866     |  67   |
| reasoning | AverageAccuracy |       arc       |                 ARC-Easy                 |    0.9661     |  59   |
|   exam    | AverageAccuracy |      iquiz      |                    IQ                    |    0.8182     |  33   |
| reasoning | AverageAccuracy |       arc       |              ARC-Challenge               |      1.0      |  31   |
|   math    |  AveragePass@1  |     aime24      |                 default                  |      0.8      |  30   |
|  general  | AverageAccuracy |    mmlu_pro     |                 physics                  |    0.9412     |  17   |
|   math    |  AveragePass@1  |     aime25      |                AIME2025-I                |      0.6      |  15   |
|   math    |  AveragePass@1  |     aime25      |               AIME2025-II                |    0.7333     |  15   |
|  general  | AverageAccuracy |    mmlu_pro     |                economics                 |      1.0      |  11   |
|  general  | AverageAccuracy |    mmlu_pro     |                   math                   |      1.0      |  10   |
|  general  | AverageAccuracy |    mmlu_pro     |                   law                    |     0.625     |   8   |
|  general  | AverageAccuracy |    mmlu_pro     |                  other                   |     0.875     |   8   |
|  general  | AverageAccuracy |    mmlu_pro     |                 business                 |    0.8571     |   7   |
|  general  | AverageAccuracy |    mmlu_pro     |                psychology                |    0.7143     |   7   |
|  general  | AverageAccuracy |    mmlu_pro     |                chemistry                 |      1.0      |   6   |
|  general  | AverageAccuracy |   mmlu_redux    |                management                |      0.8      |   5   |
|  general  | AverageAccuracy |    mmlu_pro     |                  health                  |      0.8      |   5   |
|   exam    | AverageAccuracy |      ceval      |              civil_servant               |      0.8      |   5   |
|   exam    | AverageAccuracy |      ceval      |              tax_accountant              |      0.8      |   5   |
|   exam    | AverageAccuracy |      ceval      |           discrete_mathematics           |     0.25      |   4   |
|   exam    | AverageAccuracy |      ceval      |            college_economics             |     0.75      |   4   |
|  general  | AverageAccuracy |    mmlu_pro     |                 biology                  |     0.75      |   4   |
|  general  | AverageAccuracy |   mmlu_redux    |        high_school_macroeconomics        |      1.0      |   4   |
|  general  | AverageAccuracy |   mmlu_redux    |              jurisprudence               |     0.75      |   4   |
|  general  | AverageAccuracy |   mmlu_redux    |        high_school_world_history         |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |         high_school_mathematics          |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |             security_studies             |    0.6667     |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |          professional_medicine           |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |   high_school_government_and_politics    |      1.0      |   3   |
|  general  | AverageAccuracy |    mmlu_pro     |                philosophy                |    0.3333     |   3   |
|   exam    | AverageAccuracy |      ceval      |           high_school_physics            |      1.0      |   3   |
|   exam    | AverageAccuracy |      ceval      |           high_school_politics           |      1.0      |   3   |
|   exam    | AverageAccuracy |      ceval      |         business_administration          |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |          teacher_qualification           |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |             abstract_algebra             |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |                astronomy                 |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |                 anatomy                  |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_politics          |      1.0      |   3   |
|   exam    | AverageAccuracy |      ceval      |        probability_and_statistics        |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |                physician                 |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |            mao_zedong_thought            |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            college_chemistry             |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            metrology_engineer            |      0.5      |   2   |
|   exam    | AverageAccuracy |      ceval      | environmental_impact_assessment_engineer |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           electrical_engineer            |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_psychology          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |        high_school_microeconomics        |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |         professional_accounting          |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |         professional_psychology          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |                nutrition                 |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |                philosophy                |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_statistics          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_us_history          |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |             public_relations             |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |            us_foreign_policy             |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_geography           |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          elementary_mathematics          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |             college_physics              |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |        middle_school_mathematics         |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |             plant_protection             |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_biology           |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           high_school_biology            |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           high_school_history            |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            legal_professional            |      0.5      |   2   |
|   exam    | AverageAccuracy |      ceval      |         high_school_mathematics          |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |     chinese_language_and_literature      |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            clinical_medicine             |      0.5      |   2   |
|   exam    | AverageAccuracy |      ceval      |                accountant                |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |               art_studies                |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |          computer_architecture           |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            education_science             |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           college_programming            |      1.0      |   2   |
|  general  | AverageAccuracy |    mmlu_pro     |               engineering                |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |            clinical_knowledge            |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |               econometrics               |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |           college_mathematics            |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |         college_computer_science         |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |           high_school_biology            |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |               formal_logic               |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |         urban_and_rural_planner          |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |              sports_science              |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |                 marxism                  |      0.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_physics           |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |          modern_chinese_history          |      0.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_history           |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |                   law                    |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |    ideological_and_moral_cultivation     |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |                  logic                   |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |              basic_medicine              |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |           advanced_mathematics           |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |         middle_school_chemistry          |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |         middle_school_geography          |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |             business_ethics              |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |       high_school_european_history       |      0.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_chemistry           |      0.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |          electrical_engineering          |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |               global_facts               |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |             college_medicine             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            college_chemistry             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            computer_security             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            conceptual_physics            |      1.0      |   1   |
|  general  | AverageAccuracy |    mmlu_pro     |                 history                  |      0.0      |   1   |
|  general  | AverageAccuracy |    mmlu_pro     |             computer science             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            international_law             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |                marketing                 |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |                prehistory                |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |              miscellaneous               |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |                sociology                 |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |             professional_law             |      1.0      |   1   |
+-----------+-----------------+-----------------+------------------------------------------+---------------+-------+
2025-05-24 17:34:35,915 - evalscope - INFO - dataset_level Report:
+-----------+-----------------+-----------------+---------------+-------+
| task_type |     metric      |  dataset_name   | average_score | count |
+-----------+-----------------+-----------------+---------------+-------+
|   exam    | AverageAccuracy |      iquiz      |     0.73      |  100  |
|   code    |     Pass@1      | live_code_bench |      0.0      |  90   |
|   exam    | AverageAccuracy |      ceval      |    0.8333     |  90   |
|  general  | AverageAccuracy |    mmlu_pro     |    0.8556     |  90   |
|  general  | AverageAccuracy |   mmlu_redux    |    0.8667     |  90   |
| knowledge |  AveragePass@1  |      gpqa       |    0.6333     |  90   |
|   math    | AverageAccuracy |      gsm8k      |    0.9444     |  90   |
| reasoning | AverageAccuracy |    hellaswag    |    0.7889     |  90   |
| reasoning | AverageAccuracy |       arc       |    0.9778     |  90   |
|   math    |  AveragePass@1  |     aime25      |    0.6667     |  30   |
|   math    |  AveragePass@1  |     aime24      |      0.8      |  30   |
+-----------+-----------------+-----------------+---------------+-------+
2025-05-24 17:34:35,915 - evalscope - INFO - task_level Report:
+-----------+-----------------+---------------+-------+
| task_type |     metric      | average_score | count |
+-----------+-----------------+---------------+-------+
|   exam    | AverageAccuracy |    0.7789     |  190  |
|  general  | AverageAccuracy |    0.8611     |  180  |
| reasoning | AverageAccuracy |    0.8833     |  180  |
|   code    |     Pass@1      |      0.0      |  90   |
| knowledge |  AveragePass@1  |    0.6333     |  90   |
|   math    | AverageAccuracy |    0.9444     |  90   |
|   math    |  AveragePass@1  |    0.7333     |  60   |
+-----------+-----------------+---------------+-------+
2025-05-24 17:34:35,915 - evalscope - INFO - tag_level Report:
+------+-----------------+---------------+-------+
| tags |     metric      | average_score | count |
+------+-----------------+---------------+-------+
|  en  | AverageAccuracy |    0.8867     |  450  |
|  zh  | AverageAccuracy |    0.7789     |  190  |
|  en  |  AveragePass@1  |    0.6733     |  150  |
|  en  |     Pass@1      |      0.0      |  90   |
+------+-----------------+---------------+-------+
2025-05-24 17:34:35,916 - evalscope - INFO - category_level Report:
+-----------+--------------+-----------------+---------------+-------+
| category0 |  category1   |     metric      | average_score | count |
+-----------+--------------+-----------------+---------------+-------+
|    mix    |   Chinese    | AverageAccuracy |    0.7789     |  190  |
|    mix    |  reasoning   | AverageAccuracy |    0.8833     |  180  |
|    mix    |   general    | AverageAccuracy |    0.8611     |  180  |
|    mix    | Math&Science |  AveragePass@1  |    0.6733     |  150  |
|    mix    | Math&Science | AverageAccuracy |    0.9444     |  90   |
|    mix    |     code     |     Pass@1      |      0.0      |  90   |
+-----------+--------------+-----------------+---------------+-------+
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\anaconda3\envs\evalscope\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\ProgramData\anaconda3\envs\evalscope\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] 句柄无效。
Getting reviews:  68%|██████▊   | 601/880 [00:20<00:09, 29.24it/s]2025-05-24 17:34:38,938 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
2025-05-24 17:34:38,962 - evalscope - INFO - Set resume from ./outputs/20250524_152506
2025-05-24 17:34:43,565 - evalscope - INFO - Loading dataset from C:\Users\Administrator\PycharmProjects\llmEval\EvalScope\dataset\weighted_mixed_data.jsonl
2025-05-24 17:34:48,648 - evalscope - WARNING - Output type is set to logits. This is not supported for service evaluation. Setting output type to generation by default.
2025-05-24 17:34:53,776 - evalscope - INFO - Dump task config to ./outputs/20250524_152506\configs\task_config_7fd2da.yaml
2025-05-24 17:34:53,778 - evalscope - INFO - {
    "model": "Qwen3-14B-AWQ",
    "model_id": "Qwen3-14B-AWQ",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "data_collection"
    ],
    "dataset_args": {
        "data_collection": {
            "dataset_id": "C:\\Users\\Administrator\\PycharmProjects\\llmEval\\EvalScope\\dataset\\weighted_mixed_data.jsonl",
            "filters": {
                "remove_until": "</think>"
            }
        }
    },
    "dataset_dir": "C:\\Users\\Administrator\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 30000,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20,
        "n": 1
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": null,
    "eval_batch_size": 15,
    "mem_cache": false,
    "use_cache": "./outputs/20250524_152506",
    "work_dir": "./outputs/20250524_152506",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8889/v1/chat/completions",
    "api_key": "EMPTY",
    "timeout": 60000,
    "stream": true,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
2025-05-24 17:34:54,629 - evalscope - INFO - Reuse from ./outputs/20250524_152506\predictions\Qwen3-14B-AWQ\weighted_mixed_data.jsonl. Loaded 880 samples, remain 0 samples.
Getting answers: 0it [00:00, ?it/s]
2025-05-24 17:34:54,630 - evalscope - INFO - use_cache=./outputs/20250524_152506, reloading the review file: ./outputs/20250524_152506\reviews\Qwen3-14B-AWQ
Getting reviews: 100%|██████████| 880/880 [00:00<00:00, 880483.66it/s]
Getting scores: 100%|██████████| 880/880 [00:00<00:00, 440031.89it/s]
2025-05-24 17:34:55,752 - evalscope - INFO - subset_level Report:
+-----------+-----------------+-----------------+------------------------------------------+---------------+-------+
| task_type |     metric      |  dataset_name   |               subset_name                | average_score | count |
+-----------+-----------------+-----------------+------------------------------------------+---------------+-------+
|   code    |     Pass@1      | live_code_bench |                  v5_v6                   |      0.0      |  90   |
|   math    | AverageAccuracy |      gsm8k      |                   main                   |    0.9444     |  90   |
| reasoning | AverageAccuracy |    hellaswag    |                 default                  |    0.7889     |  90   |
| knowledge |  AveragePass@1  |      gpqa       |               gpqa_diamond               |    0.6333     |  90   |
|   exam    | AverageAccuracy |      iquiz      |                    EQ                    |    0.6866     |  67   |
| reasoning | AverageAccuracy |       arc       |                 ARC-Easy                 |    0.9661     |  59   |
|   exam    | AverageAccuracy |      iquiz      |                    IQ                    |    0.8182     |  33   |
| reasoning | AverageAccuracy |       arc       |              ARC-Challenge               |      1.0      |  31   |
|   math    |  AveragePass@1  |     aime24      |                 default                  |      0.8      |  30   |
|  general  | AverageAccuracy |    mmlu_pro     |                 physics                  |    0.9412     |  17   |
|   math    |  AveragePass@1  |     aime25      |                AIME2025-I                |      0.6      |  15   |
|   math    |  AveragePass@1  |     aime25      |               AIME2025-II                |    0.7333     |  15   |
|  general  | AverageAccuracy |    mmlu_pro     |                economics                 |      1.0      |  11   |
|  general  | AverageAccuracy |    mmlu_pro     |                   math                   |      1.0      |  10   |
|  general  | AverageAccuracy |    mmlu_pro     |                   law                    |     0.625     |   8   |
|  general  | AverageAccuracy |    mmlu_pro     |                  other                   |     0.875     |   8   |
|  general  | AverageAccuracy |    mmlu_pro     |                 business                 |    0.8571     |   7   |
|  general  | AverageAccuracy |    mmlu_pro     |                psychology                |    0.7143     |   7   |
|  general  | AverageAccuracy |    mmlu_pro     |                chemistry                 |      1.0      |   6   |
|  general  | AverageAccuracy |   mmlu_redux    |                management                |      0.8      |   5   |
|  general  | AverageAccuracy |    mmlu_pro     |                  health                  |      0.8      |   5   |
|   exam    | AverageAccuracy |      ceval      |              civil_servant               |      0.8      |   5   |
|   exam    | AverageAccuracy |      ceval      |              tax_accountant              |      0.8      |   5   |
|   exam    | AverageAccuracy |      ceval      |           discrete_mathematics           |     0.25      |   4   |
|   exam    | AverageAccuracy |      ceval      |            college_economics             |     0.75      |   4   |
|  general  | AverageAccuracy |    mmlu_pro     |                 biology                  |     0.75      |   4   |
|  general  | AverageAccuracy |   mmlu_redux    |        high_school_macroeconomics        |      1.0      |   4   |
|  general  | AverageAccuracy |   mmlu_redux    |              jurisprudence               |     0.75      |   4   |
|  general  | AverageAccuracy |   mmlu_redux    |        high_school_world_history         |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |         high_school_mathematics          |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |             security_studies             |    0.6667     |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |          professional_medicine           |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |   high_school_government_and_politics    |      1.0      |   3   |
|  general  | AverageAccuracy |    mmlu_pro     |                philosophy                |    0.3333     |   3   |
|   exam    | AverageAccuracy |      ceval      |           high_school_physics            |      1.0      |   3   |
|   exam    | AverageAccuracy |      ceval      |           high_school_politics           |      1.0      |   3   |
|   exam    | AverageAccuracy |      ceval      |         business_administration          |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |          teacher_qualification           |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |             abstract_algebra             |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |                astronomy                 |      1.0      |   3   |
|  general  | AverageAccuracy |   mmlu_redux    |                 anatomy                  |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_politics          |      1.0      |   3   |
|   exam    | AverageAccuracy |      ceval      |        probability_and_statistics        |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |                physician                 |    0.6667     |   3   |
|   exam    | AverageAccuracy |      ceval      |            mao_zedong_thought            |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            college_chemistry             |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            metrology_engineer            |      0.5      |   2   |
|   exam    | AverageAccuracy |      ceval      | environmental_impact_assessment_engineer |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           electrical_engineer            |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_psychology          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |        high_school_microeconomics        |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |         professional_accounting          |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |         professional_psychology          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |                nutrition                 |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |                philosophy                |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_statistics          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_us_history          |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |             public_relations             |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |            us_foreign_policy             |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_geography           |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |          elementary_mathematics          |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |             college_physics              |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |        middle_school_mathematics         |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |             plant_protection             |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_biology           |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           high_school_biology            |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           high_school_history            |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            legal_professional            |      0.5      |   2   |
|   exam    | AverageAccuracy |      ceval      |         high_school_mathematics          |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |     chinese_language_and_literature      |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            clinical_medicine             |      0.5      |   2   |
|   exam    | AverageAccuracy |      ceval      |                accountant                |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |               art_studies                |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |          computer_architecture           |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |            education_science             |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |           college_programming            |      1.0      |   2   |
|  general  | AverageAccuracy |    mmlu_pro     |               engineering                |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |            clinical_knowledge            |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |               econometrics               |      0.5      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |           college_mathematics            |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |         college_computer_science         |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |           high_school_biology            |      1.0      |   2   |
|  general  | AverageAccuracy |   mmlu_redux    |               formal_logic               |      1.0      |   2   |
|   exam    | AverageAccuracy |      ceval      |         urban_and_rural_planner          |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |              sports_science              |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |                 marxism                  |      0.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_physics           |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |          modern_chinese_history          |      0.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |          middle_school_history           |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |                   law                    |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |    ideological_and_moral_cultivation     |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |                  logic                   |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |              basic_medicine              |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |           advanced_mathematics           |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |         middle_school_chemistry          |      1.0      |   1   |
|   exam    | AverageAccuracy |      ceval      |         middle_school_geography          |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |             business_ethics              |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |       high_school_european_history       |      0.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |          high_school_chemistry           |      0.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |          electrical_engineering          |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |               global_facts               |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |             college_medicine             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            college_chemistry             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            computer_security             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            conceptual_physics            |      1.0      |   1   |
|  general  | AverageAccuracy |    mmlu_pro     |                 history                  |      0.0      |   1   |
|  general  | AverageAccuracy |    mmlu_pro     |             computer science             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |            international_law             |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |                marketing                 |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |                prehistory                |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |              miscellaneous               |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |                sociology                 |      1.0      |   1   |
|  general  | AverageAccuracy |   mmlu_redux    |             professional_law             |      1.0      |   1   |
+-----------+-----------------+-----------------+------------------------------------------+---------------+-------+
2025-05-24 17:34:55,754 - evalscope - INFO - dataset_level Report:
+-----------+-----------------+-----------------+---------------+-------+
| task_type |     metric      |  dataset_name   | average_score | count |
+-----------+-----------------+-----------------+---------------+-------+
|   exam    | AverageAccuracy |      iquiz      |     0.73      |  100  |
|   code    |     Pass@1      | live_code_bench |      0.0      |  90   |
|   exam    | AverageAccuracy |      ceval      |    0.8333     |  90   |
|  general  | AverageAccuracy |    mmlu_pro     |    0.8556     |  90   |
|  general  | AverageAccuracy |   mmlu_redux    |    0.8667     |  90   |
| knowledge |  AveragePass@1  |      gpqa       |    0.6333     |  90   |
|   math    | AverageAccuracy |      gsm8k      |    0.9444     |  90   |
| reasoning | AverageAccuracy |    hellaswag    |    0.7889     |  90   |
| reasoning | AverageAccuracy |       arc       |    0.9778     |  90   |
|   math    |  AveragePass@1  |     aime25      |    0.6667     |  30   |
|   math    |  AveragePass@1  |     aime24      |      0.8      |  30   |
+-----------+-----------------+-----------------+---------------+-------+
2025-05-24 17:34:55,754 - evalscope - INFO - task_level Report:
+-----------+-----------------+---------------+-------+
| task_type |     metric      | average_score | count |
+-----------+-----------------+---------------+-------+
|   exam    | AverageAccuracy |    0.7789     |  190  |
|  general  | AverageAccuracy |    0.8611     |  180  |
| reasoning | AverageAccuracy |    0.8833     |  180  |
|   code    |     Pass@1      |      0.0      |  90   |
| knowledge |  AveragePass@1  |    0.6333     |  90   |
|   math    | AverageAccuracy |    0.9444     |  90   |
|   math    |  AveragePass@1  |    0.7333     |  60   |
+-----------+-----------------+---------------+-------+
2025-05-24 17:34:55,754 - evalscope - INFO - tag_level Report:
+------+-----------------+---------------+-------+
| tags |     metric      | average_score | count |
+------+-----------------+---------------+-------+
|  en  | AverageAccuracy |    0.8867     |  450  |
|  zh  | AverageAccuracy |    0.7789     |  190  |
|  en  |  AveragePass@1  |    0.6733     |  150  |
|  en  |     Pass@1      |      0.0      |  90   |
+------+-----------------+---------------+-------+
2025-05-24 17:34:55,755 - evalscope - INFO - category_level Report:
+-----------+--------------+-----------------+---------------+-------+
| category0 |  category1   |     metric      | average_score | count |
+-----------+--------------+-----------------+---------------+-------+
|    mix    |   Chinese    | AverageAccuracy |    0.7789     |  190  |
|    mix    |  reasoning   | AverageAccuracy |    0.8833     |  180  |
|    mix    |   general    | AverageAccuracy |    0.8611     |  180  |
|    mix    | Math&Science |  AveragePass@1  |    0.6733     |  150  |
|    mix    | Math&Science | AverageAccuracy |    0.9444     |  90   |
|    mix    |     code     |     Pass@1      |      0.0      |  90   |
+-----------+--------------+-----------------+---------------+-------+
Getting reviews:  68%|██████▊   | 601/880 [00:39<00:09, 29.24it/s]Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\anaconda3\envs\evalscope\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\ProgramData\anaconda3\envs\evalscope\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] 句柄无效。
Getting reviews:  68%|██████▊   | 602/880 [00:40<00:22, 12.27it/s]2025-05-24 17:34:58,919 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
```
后面一直循环打印report和报错OSError: [WinError 6] 句柄无效

## 运行环境

- 操作系统：windows11  26100.4061
- Python版本：Python 3.10.16

## 其他信息

不使用use_cache的情况下，Getting answers 完成后，会一直重复Getting answers 


看起来可能是在测试live_code_bench时遇到了问题，单独测试live_code_bench试试呢

是的，live code bench的问题
```bash
2025-05-27 19:37:41,853 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
2025-05-27 19:37:46,194 - evalscope - INFO - Loading dataset from ..\dataset\live_code_bench.jsonl
2025-05-27 19:37:47,369 - evalscope - INFO - Dump task config to ./outputs\20250527_193741\configs\task_config_48e370.yaml
2025-05-27 19:37:47,381 - evalscope - INFO - {
    "model": "Qwen/Qwen3-1.7B",
    "model_id": "Qwen3-1.7B",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "data_collection"
    ],
    "dataset_args": {
        "data_collection": {
            "dataset_id": "..\\dataset\\live_code_bench.jsonl",
            "filters": {
                "remove_until": "</think>"
            }
        }
    },
    "dataset_dir": "C:\\Users\\Administrator\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 30000,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20,
        "n": 1
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 100,
    "eval_batch_size": 20,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250527_193741",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8000/v1/chat/completions",
    "api_key": "EMPTY",
    "timeout": 60000,
    "stream": true,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
Getting answers: 100%|██████████| 4/4 [04:22<00:00, 65.74s/it]
Getting reviews:   0%|          | 0/4 [00:00<?, ?it/s]2025-05-27 19:42:12,936 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
2025-05-27 19:42:17,910 - evalscope - INFO - Loading dataset from ..\dataset\live_code_bench.jsonl
2025-05-27 19:42:19,598 - evalscope - INFO - Dump task config to ./outputs\20250527_194212\configs\task_config_af4704.yaml
2025-05-27 19:42:19,602 - evalscope - INFO - {
    "model": "Qwen/Qwen3-1.7B",
    "model_id": "Qwen3-1.7B",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "data_collection"
    ],
    "dataset_args": {
        "data_collection": {
            "dataset_id": "..\\dataset\\live_code_bench.jsonl",
            "filters": {
                "remove_until": "</think>"
            }
        }
    },
    "dataset_dir": "C:\\Users\\Administrator\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 30000,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20,
        "n": 1
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 100,
    "eval_batch_size": 20,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250527_194212",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8000/v1/chat/completions",
    "api_key": "EMPTY",
    "timeout": 60000,
    "stream": true,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
```
Getting reviews中断，然后重复执行

我也遇到了相同问题，是否有解决办法?

这个问题暂时不好解决，live code bench评测是在本地执行模型生成的代码，这个部分评测代码是参考 https://github.com/LiveCodeBench/LiveCodeBench 官方，但也可能存在一些问题。后续期望是启动一个代码执行的服务来进行评测。

使用ubuntu系统或者windows上的WSL是否可以避免？

@AInseven 可以尝试一下ubuntu系统，我这里运行正常


这个问题是LCB评测代码中使用了windows上不支持信号机制`signal.alarm()`造成的，暂时无法解决，建议使用ubuntu等linux系统进行评测

目前已支持在沙箱环境中运行代码，请参考这个文档：https://evalscope.readthedocs.io/zh-cn/latest/user_guides/sandbox.html

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！