Repository: modelscope/evalscope
Issue: 离线评测cmmlu时，把csv第一行的表头也当成问题
Root cause: Validation Gap

系统：windows11
Python 3.10.10
evalscope 0.16.3

我用cmd执行：evalscope eval --model Qwen2.5-1.5B-Instruct-Q4_K_S-gguf --api-url http://localhost:8080/v1/chat/completions --api-key empty --eval-type service --datasets cmmlu --dataset-args {\"cmmlu\":{\"local_path\":\"C:/Users/yerke/Desktop/aipc_new/aipc/resources/datasets/data/cmmlu\"}} --limit 1

离线数据是从https://modelscope.oss-cn-beijing.aliyuncs.com/open_data/benchmark/data.zip提取的
![Image](https://github.com/user-attachments/assets/586ebc72-1635-45f2-8fac-a0734961efc4)

编译的predictions是这样的
![Image](https://github.com/user-attachments/assets/e813ce00-1628-4fd0-bd9f-311469f80434)

日志大致是这样打印的：
C:\Users\yerke>evalscope eval --model Qwen2.5-1.5B-Instruct-Q4_K_S-gguf --api-url http://localhost:8080/v1/chat/completions --api-key empty --eval-type service --datasets cmmlu --dataset-args {\"cmmlu\":{\"local_path\":\"C:/Users/yerke/Desktop/aipc_new/aipc/resources/datasets/data/cmmlu\"}} --limit 1
2025-06-25 11:18:20,760 - evalscope - INFO - Args: Task config is provided with CommandLine type.
2025-06-25 11:18:28,039 - evalscope - INFO - Dump task config to ./outputs\20250625_111820\configs\task_config_359c8a.yaml
2025-06-25 11:18:28,044 - evalscope - INFO - {
    "model": "Qwen2.5-1.5B-Instruct-Q4_K_S-gguf",
    "model_id": "Qwen2.5-1.5B-Instruct-Q4_K_S-gguf",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "cmmlu"
    ],
    "dataset_args": {
        "cmmlu": {
            "name": "cmmlu",
            "dataset_id": "C:/Users/yerke/Desktop/aipc_new/aipc/resources/datasets/data/cmmlu",
            "model_adapter": "generation",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "agronomy",
                "anatomy",
                "ancient_chinese",
                "arts",
                "astronomy",
                "business_ethics",
                "chinese_civil_service_exam",
                "chinese_driving_rule",
                "chinese_food_culture",
                "chinese_foreign_policy",
                "chinese_history",
                "chinese_literature",
                "chinese_teacher_qualification",
                "college_actuarial_science",
                "college_education",
                "college_engineering_hydrology",
                "college_law",
                "college_mathematics",
                "college_medical_statistics",
                "clinical_knowledge",
                "college_medicine",
                "computer_science",
                "computer_security",
                "conceptual_physics",
                "construction_project_management",
                "economics",
                "education",
                "elementary_chinese",
                "elementary_commonsense",
                "elementary_information_and_technology",
                "electrical_engineering",
                "elementary_mathematics",
                "ethnology",
                "food_science",
                "genetics",
                "global_facts",
                "high_school_biology",
                "high_school_chemistry",
                "high_school_geography",
                "high_school_mathematics",
                "high_school_physics",
                "high_school_politics",
                "human_sexuality",
                "international_law",
                "journalism",
                "jurisprudence",
                "legal_and_moral_basis",
                "logical",
                "machine_learning",
                "management",
                "marketing",
                "marxist_theory",
                "modern_chinese",
                "nutrition",
                "philosophy",
                "professional_accounting",
                "professional_law",
                "professional_medicine",
                "professional_psychology",
                "public_relations",
                "security_study",
                "sociology",
                "sports_science",
                "traditional_chinese_medicine",
                "virology",
                "world_history",
                "world_religions"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 5,
            "few_shot_random": false,
            "train_split": "dev",
            "eval_split": "test",
            "prompt_template": "以下是关于{subset_name}的单项选择题，请给出正确答案的选项。你的回答的最后一行应该是这样的格式：“答案：LETTER”（不带引号），其中 LETTER 是 A、B、C、D 中的一个。\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "C-MMLU",
            "description": "C-MMLU is a benchmark designed to evaluate the performance of AI models on Chinese language tasks, including reading comprehension, text classification, and more.",
            "tags": [
                "Knowledge",
                "MCQ",
                "Chinese"
            ],
            "filters": null,
            "extra_params": {}
        }
    },
    "dataset_dir": "C:\\Users\\yerke\\.cache\\modelscope\\hub\\datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 2048,
        "temperature": 0.0
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 1,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs\\20250625_111820",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://localhost:8080/v1/chat/completions",
    "api_key": "empty",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-06-25 11:18:28,044 - evalscope - INFO - Start evaluating on dataset C:/Users/yerke/Desktop/aipc_new/aipc/resources/datasets/data/cmmlu
2025-06-25 11:18:28,044 - evalscope - INFO - Loading dataset from local disk: C:/Users/yerke/Desktop/aipc_new/aipc/resources/datasets/data/cmmlu
2025-06-25 11:18:28,739 - evalscope - INFO - Use settings: > few_shot_num: 5, > few_shot_split: dev, > target_eval_split: test
Predicting(agronomy): 100%|██████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.41s/it]
2025-06-25 11:18:31,489 - evalscope - INFO - Dump predictions to ./outputs\20250625_111820\predictions\Qwen2.5-1.5B-Instruct-Q4_K_S-gguf\cmmlu_agronomy.jsonl.
Reviewing(agronomy): 100%|█████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1004.38it/s]
Predicting(anatomy): 100%|███████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  9.17it/s]
2025-06-25 11:18:31,608 - evalscope - INFO - Dump predictions to ./outputs\20250625_111820\predictions\Qwen2.5-1.5B-Instruct-Q4_K_S-gguf\cmmlu_anatomy.jsonl.
Reviewing(anatomy): 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1001.74it/s]
Predicting(ancient_chinese): 100%|███████████████████████████████████████████████████████| 1/1 [00:00<00:00,  8.46it/s]
2025-06-25 11:18:31,730 - evalscope - INFO - Dump predictions to ./outputs\20250625_111820\predictions\Qwen2.5-1.5B-Instruct-Q4_K_S-gguf\cmmlu_ancient_chinese.jsonl.
Reviewing(ancient_chinese): 100%|████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Predicting(arts): 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  9.37it/s]
2025-06-25 11:18:31,841 - evalscope - INFO - Dump predictions to ./outputs\20250625_111820\predictions\Qwen2.5-1.5B-Instruct-Q4_K_S-gguf\cmmlu_arts.jsonl.
我哪里操作错误了吗

我复现一下这个问题

已修复该问题，请使用main分支代码再试试

> 已修复该问题，请使用main分支代码再试试

感谢，确实修复了，但这个"": "0"是一定要有的吗
![Image](https://github.com/user-attachments/assets/5c7d3c77-7147-44d5-9d2a-626bc402e0cf)

> 但这个"": "0"是一定要有的吗

这个不影响，是原始数据集格式的问题，可以忽略

> > 但这个"": "0"是一定要有的吗
> 
> 这个不影响，是原始数据集格式的问题，可以忽略

好的，感谢感谢