Repository: modelscope/evalscope
Issue: mmlu数据集与opencompass结果差异较大
Root cause: Algorithmic Error

EvalScope 0.17.0
opencompass 0.4.2

测试DeepSeek-R1-AWQ在mmlu_gen数据集上的accuracy
配置文件如下
```
from evalscope import TaskConfig, run_task
#  start the task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    stream=True,
    timeout=60000,
    model='/models/DeepSeek-R1-awq',
    api_url='http://0.0.0.0:8000/v1/chat/completions',
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,
    eval_batch_size=64,
    generation_config={
        'max_tokens': 16384,  # avoid exceed max length
        'temperature': 0.6,
        'top_p': 0.95,
        'n': 1,  # num of responses per sample, note that lmdeploy only supports n=1
    },
    datasets=['mmlu'],
    dataset_args={"mmlu": {"local_path": "/home/evalscope-main/data/mmlu"}}
)
run_task(task_cfg=task_cfg)
```

eval_log.log如下
```
2025-07-12 21:54:15,757 - evalscope - INFO - Dump task config to ./outputs/20250712_215414/configs/task_config_58323d.yaml
2025-07-12 21:54:15,763 - evalscope - INFO - {
    "model": "/models/DeepSeek-R1-awq",
    "model_id": "DeepSeek-R1-awq",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "mmlu"
    ],
    "dataset_args": {
        "mmlu": {
            "name": "mmlu",
            "dataset_id": "modelscope/mmlu",
            "model_adapter": "generation",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "high_school_european_history",
                "business_ethics",
                "clinical_knowledge",
                "medical_genetics",
                "high_school_us_history",
                "high_school_physics",
                "high_school_world_history",
                "virology",
                "high_school_microeconomics",
                "econometrics",
                "college_computer_science",
                "high_school_biology",
                "abstract_algebra",
                "professional_accounting",
                "philosophy",
                "professional_medicine",
                "nutrition",
                "global_facts",
                "machine_learning",
                "security_studies",
                "public_relations",
                "professional_psychology",
                "prehistory",
                "anatomy",
                "human_sexuality",
                "college_medicine",
                "high_school_government_and_politics",
                "college_chemistry",
                "logical_fallacies",
                "high_school_geography",
                "elementary_mathematics",
                "human_aging",
                "college_mathematics",
                "high_school_psychology",
                "formal_logic",
                "high_school_statistics",
                "international_law",
                "high_school_mathematics",
                "high_school_computer_science",
                "conceptual_physics",
                "miscellaneous",
                "high_school_chemistry",
                "marketing",
                "professional_law",
                "management",
                "college_physics",
                "jurisprudence",
                "world_religions",
                "sociology",
                "us_foreign_policy",
                "high_school_macroeconomics",
                "computer_security",
                "moral_scenarios",
                "moral_disputes",
                "electrical_engineering",
                "astronomy",
                "college_biology"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 5,
            "few_shot_random": false,
            "train_split": "train",
            "eval_split": "test",
            "prompt_template": "Answer the following multiple choice question about {subset_name}. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.\n\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "MMLU",
            "description": "The MMLU (Massive Multitask Language Understanding) benchmark is a comprehensive evaluation suite designed to assess the performance of language models across a wide range of subjects and tasks. It includes multiple-choice questions from various domains, such as history, science, mathematics, and more, providing a robust measure of a model's understanding and reasoning capabilities.",
            "tags": [
                "Knowledge",
                "MCQ"
            ],
            "filters": null,
            "extra_params": {}
        }
    },
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 16384,
        "temperature": 0.6,
        "top_p": 0.95,
        "n": 1
    },
    "eval_type": "service",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": null,
    "eval_batch_size": 64,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs/20250712_215414",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://0.0.0.0:8000/v1/chat/completions",
    "api_key": "EMPTY",
    "timeout": 60000,
    "stream": true,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-07-12 21:54:15,763 - evalscope - INFO - Start evaluating on dataset modelscope/mmlu
2025-07-12 21:54:15,763 - evalscope - INFO - Loading dataset from hub: modelscope/mmlu
2025-07-12 21:54:16,117 - evalscope - INFO - Loading dataset: dataset_name: modelscope/mmlu > subsets: ['high_school_european_history', 'business_ethics', 'clinical_knowledge', 'medical_genetics', 'high_school_us_history', 'high_school_physics', 'high_school_world_history', 'virology', 'high_school_microeconomics', 'econometrics', 'college_computer_science', 'high_school_biology', 'abstract_algebra', 'professional_accounting', 'philosophy', 'professional_medicine', 'nutrition', 'global_facts', 'machine_learning', 'security_studies', 'public_relations', 'professional_psychology', 'prehistory', 'anatomy', 'human_sexuality', 'college_medicine', 'high_school_government_and_politics', 'college_chemistry', 'logical_fallacies', 'high_school_geography', 'elementary_mathematics', 'human_aging', 'college_mathematics', 'high_school_psychology', 'formal_logic', 'high_school_statistics', 'international_law', 'high_school_mathematics', 'high_school_computer_science', 'conceptual_physics', 'miscellaneous', 'high_school_chemistry', 'marketing', 'professional_law', 'management', 'college_physics', 'jurisprudence', 'world_religions', 'sociology', 'us_foreign_policy', 'high_school_macroeconomics', 'computer_security', 'moral_scenarios', 'moral_disputes', 'electrical_engineering', 'astronomy', 'college_biology']
2025-07-12 22:04:27,095 - evalscope - INFO - Use settings: > few_shot_num: 5, > few_shot_split: train, > target_eval_split: test
2025-07-12 22:30:27,339 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_european_history.jsonl.
2025-07-12 22:41:13,945 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_business_ethics.jsonl.
2025-07-12 23:04:47,224 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_clinical_knowledge.jsonl.
2025-07-12 23:20:24,697 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_medical_genetics.jsonl.
2025-07-12 23:44:52,918 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_us_history.jsonl.
2025-07-13 00:19:48,494 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_physics.jsonl.
2025-07-13 00:46:56,278 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_world_history.jsonl.
2025-07-13 01:09:43,391 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_virology.jsonl.
2025-07-13 01:32:34,555 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_microeconomics.jsonl.
2025-07-13 03:06:32,510 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_econometrics.jsonl.
2025-07-13 03:36:07,933 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_college_computer_science.jsonl.
2025-07-13 04:06:36,603 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_biology.jsonl.
2025-07-13 04:25:50,607 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_abstract_algebra.jsonl.
2025-07-13 06:42:48,485 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_professional_accounting.jsonl.
2025-07-13 07:06:02,771 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_philosophy.jsonl.
2025-07-13 07:48:08,848 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_professional_medicine.jsonl.
2025-07-13 08:56:33,005 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_nutrition.jsonl.
2025-07-13 09:10:29,411 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_global_facts.jsonl.
2025-07-13 09:41:11,565 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_machine_learning.jsonl.
2025-07-13 10:26:12,761 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_security_studies.jsonl.
2025-07-13 10:39:35,837 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_public_relations.jsonl.
2025-07-13 11:28:26,720 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_professional_psychology.jsonl.
2025-07-13 12:01:07,449 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_prehistory.jsonl.
2025-07-13 12:18:35,602 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_anatomy.jsonl.
2025-07-13 12:33:21,354 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_human_sexuality.jsonl.
2025-07-13 13:21:20,313 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_college_medicine.jsonl.
2025-07-13 13:34:11,711 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_government_and_politics.jsonl.
2025-07-13 14:26:18,047 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_college_chemistry.jsonl.
2025-07-13 14:42:47,972 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_logical_fallacies.jsonl.
2025-07-13 15:12:25,179 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_geography.jsonl.
2025-07-13 15:45:44,301 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_elementary_mathematics.jsonl.
2025-07-13 16:05:04,651 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_human_aging.jsonl.
2025-07-13 16:32:56,463 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_college_mathematics.jsonl.
2025-07-13 17:13:16,879 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_psychology.jsonl.
2025-07-13 17:51:14,151 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_formal_logic.jsonl.
2025-07-13 18:22:58,339 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_statistics.jsonl.
2025-07-13 18:37:13,679 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_international_law.jsonl.
2025-07-13 19:37:01,124 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_mathematics.jsonl.
2025-07-13 20:16:47,019 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_computer_science.jsonl.
2025-07-13 20:44:29,251 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_conceptual_physics.jsonl.
2025-07-13 21:28:15,573 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_miscellaneous.jsonl.
2025-07-13 22:18:34,256 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_chemistry.jsonl.
2025-07-13 22:40:03,211 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_marketing.jsonl.
2025-07-14 06:16:08,075 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_professional_law.jsonl.
2025-07-14 06:23:33,138 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_management.jsonl.
2025-07-14 06:48:13,510 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_college_physics.jsonl.
2025-07-14 06:59:45,994 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_jurisprudence.jsonl.
2025-07-14 07:09:46,301 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_world_religions.jsonl.
2025-07-14 07:35:22,091 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_sociology.jsonl.
2025-07-14 07:44:36,649 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_us_foreign_policy.jsonl.
2025-07-14 08:20:32,800 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_high_school_macroeconomics.jsonl.
2025-07-14 08:41:09,906 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_computer_security.jsonl.
2025-07-14 10:15:50,912 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_moral_scenarios.jsonl.
2025-07-14 10:48:04,356 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_moral_disputes.jsonl.
2025-07-14 11:22:46,371 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_electrical_engineering.jsonl.
2025-07-14 11:40:07,478 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_astronomy.jsonl.
2025-07-14 11:58:05,013 - evalscope - INFO - Dump predictions to ./outputs/20250712_215414/predictions/DeepSeek-R1-awq/mmlu_college_biology.jsonl.
2025-07-14 11:58:05,064 - evalscope - INFO - 
modelscope/mmlu report table:
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Model           | Dataset   | Metric          | Subset                              |   Num |   Score | Cat.0          |
+=================+===========+=================+=====================================+=======+=========+================+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_european_history        |   164 |  0.8598 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_us_history              |   203 |  0.9163 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_world_history           |   236 |  0.9449 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | philosophy                          |   310 |  0.4581 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | prehistory                          |   323 |  0.3498 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | logical_fallacies                   |   162 |  0.6975 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | formal_logic                        |   125 |  0.376  | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | international_law                   |   120 |  0.55   | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_law                    |  1533 |  0.7671 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | jurisprudence                       |   107 |  0.4206 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | world_religions                     |   170 |  0.7235 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | moral_scenarios                     |   894 |  0.7886 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | moral_disputes                      |   345 |  0.5101 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | business_ethics                     |    99 |  0.6667 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | clinical_knowledge                  |   264 |  0.8295 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | medical_genetics                    |    99 |  0.4242 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | virology                            |   165 |  0.3515 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_accounting             |   281 |  0.6441 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_medicine               |   271 |  0.9077 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | nutrition                           |   305 |  0.3934 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | global_facts                        |    99 |  0.5657 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | anatomy                             |   134 |  0.6343 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_medicine                    |   172 |  0.5872 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | human_aging                         |   222 |  0.5766 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | miscellaneous                       |   782 |  0.6637 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | marketing                           |   233 |  0.4421 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | management                          |   102 |  0.2647 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_physics                 |   150 |  0.7333 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_computer_science            |    99 |  0.6263 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_biology                 |   309 |  0.4757 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | abstract_algebra                    |    99 |  0.7172 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | machine_learning                    |   111 |  0.3784 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_chemistry                   |    99 |  0.5253 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | elementary_mathematics              |   377 |  0.4775 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_mathematics                 |    99 |  0.7879 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_statistics              |   215 |  0.5721 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_mathematics             |   269 |  0.5316 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_computer_science        |    99 |  0.3232 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | conceptual_physics                  |   234 |  0.4274 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_chemistry               |   202 |  0.2871 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_physics                     |   101 |  0.5545 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | computer_security                   |    99 |  0.6162 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | electrical_engineering              |   144 |  0.4514 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | astronomy                           |   151 |  0.5033 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_biology                     |   143 |  0.8951 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_microeconomics          |   237 |  0.7089 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | econometrics                        |   113 |  0.5221 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | security_studies                    |   244 |  0.5369 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | public_relations                    |   109 |  0.3945 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_psychology             |   611 |  0.5957 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | human_sexuality                     |   130 |  0.5385 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_government_and_politics |   192 |  0.9479 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_geography               |   197 |  0.2893 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_psychology              |   544 |  0.4559 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | sociology                           |   200 |  0.49   | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | us_foreign_policy                   |    99 |  0.899  | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_macroeconomics          |   389 |  0.5913 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | OVERALL                             | 13985 |  0.6099 | -              |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+ 

2025-07-14 11:58:05,064 - evalscope - INFO - Skipping report analysis (`analysis_report=False`).
2025-07-14 11:58:05,065 - evalscope - INFO - Dump report to: ./outputs/20250712_215414/reports/DeepSeek-R1-awq/mmlu.json 

2025-07-14 11:58:05,065 - evalscope - INFO - Evaluation finished on modelscope/mmlu
2025-07-14 11:58:05,136 - evalscope - INFO - Overall report table: 
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Model           | Dataset   | Metric          | Subset                              |   Num |   Score | Cat.0          |
+=================+===========+=================+=====================================+=======+=========+================+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_european_history        |   164 |  0.8598 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_us_history              |   203 |  0.9163 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_world_history           |   236 |  0.9449 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | philosophy                          |   310 |  0.4581 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | prehistory                          |   323 |  0.3498 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | logical_fallacies                   |   162 |  0.6975 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | formal_logic                        |   125 |  0.376  | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | international_law                   |   120 |  0.55   | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_law                    |  1533 |  0.7671 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | jurisprudence                       |   107 |  0.4206 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | world_religions                     |   170 |  0.7235 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | moral_scenarios                     |   894 |  0.7886 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | moral_disputes                      |   345 |  0.5101 | Humanities     |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | business_ethics                     |    99 |  0.6667 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | clinical_knowledge                  |   264 |  0.8295 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | medical_genetics                    |    99 |  0.4242 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | virology                            |   165 |  0.3515 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_accounting             |   281 |  0.6441 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_medicine               |   271 |  0.9077 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | nutrition                           |   305 |  0.3934 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | global_facts                        |    99 |  0.5657 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | anatomy                             |   134 |  0.6343 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_medicine                    |   172 |  0.5872 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | human_aging                         |   222 |  0.5766 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | miscellaneous                       |   782 |  0.6637 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | marketing                           |   233 |  0.4421 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | management                          |   102 |  0.2647 | Other          |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_physics                 |   150 |  0.7333 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_computer_science            |    99 |  0.6263 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_biology                 |   309 |  0.4757 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | abstract_algebra                    |    99 |  0.7172 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | machine_learning                    |   111 |  0.3784 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_chemistry                   |    99 |  0.5253 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | elementary_mathematics              |   377 |  0.4775 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_mathematics                 |    99 |  0.7879 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_statistics              |   215 |  0.5721 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_mathematics             |   269 |  0.5316 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_computer_science        |    99 |  0.3232 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | conceptual_physics                  |   234 |  0.4274 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_chemistry               |   202 |  0.2871 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_physics                     |   101 |  0.5545 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | computer_security                   |    99 |  0.6162 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | electrical_engineering              |   144 |  0.4514 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | astronomy                           |   151 |  0.5033 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | college_biology                     |   143 |  0.8951 | STEM           |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_microeconomics          |   237 |  0.7089 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | econometrics                        |   113 |  0.5221 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | security_studies                    |   244 |  0.5369 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | public_relations                    |   109 |  0.3945 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | professional_psychology             |   611 |  0.5957 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | human_sexuality                     |   130 |  0.5385 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_government_and_politics |   192 |  0.9479 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_geography               |   197 |  0.2893 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_psychology              |   544 |  0.4559 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | sociology                           |   200 |  0.49   | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | us_foreign_policy                   |    99 |  0.899  | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | high_school_macroeconomics          |   389 |  0.5913 | Social Science |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1-awq | mmlu      | AverageAccuracy | OVERALL                             | 13985 |  0.6099 | -              |
+-----------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+ 

2025-07-14 11:58:05,136 - evalscope - INFO - Finished evaluation for DeepSeek-R1-awq on ['mmlu']
2025-07-14 11:58:05,137 - evalscope - INFO - Output directory: ./outputs/20250712_215414
```

opencompass结果如下，acc均值在90左右
| dataset | version | metric | mode | vllm_deepseek_r1_671b_AWQ_int4 |
|----- | ----- | ----- | ----- | -----|
| lukaemon_mmlu_college_biology | bf6b83 | accuracy | gen | 98.61 |
| lukaemon_mmlu_college_chemistry | bf6b83 | accuracy | gen | 73.00 |
| lukaemon_mmlu_college_computer_science | bf6b83 | accuracy | gen | 92.00 |
| lukaemon_mmlu_college_mathematics | bf6b83 | accuracy | gen | 88.00 |
| lukaemon_mmlu_college_physics | bf6b83 | accuracy | gen | 96.08 |
| lukaemon_mmlu_electrical_engineering | bf6b83 | accuracy | gen | 87.59 |
| lukaemon_mmlu_astronomy | bf6b83 | accuracy | gen | 96.05 |
| lukaemon_mmlu_anatomy | bf6b83 | accuracy | gen | 91.85 |
| lukaemon_mmlu_abstract_algebra | bf6b83 | accuracy | gen | 88.00 |
| lukaemon_mmlu_machine_learning | bf6b83 | accuracy | gen | 83.93 |
| lukaemon_mmlu_clinical_knowledge | bf6b83 | accuracy | gen | 94.72 |
| lukaemon_mmlu_global_facts | bf6b83 | accuracy | gen | 68.00 |
| lukaemon_mmlu_management | bf6b83 | accuracy | gen | 91.26 |
| lukaemon_mmlu_nutrition | bf6b83 | accuracy | gen | 94.77 |
| lukaemon_mmlu_marketing | bf6b83 | accuracy | gen | 95.30 |
| lukaemon_mmlu_professional_accounting | bf6b83 | accuracy | gen | 97.87 |
| lukaemon_mmlu_high_school_geography | bf6b83 | accuracy | gen | 93.94 |
| lukaemon_mmlu_international_law | bf6b83 | accuracy | gen | 90.91 |
| lukaemon_mmlu_moral_scenarios | bf6b83 | accuracy | gen | 77.77 |
| lukaemon_mmlu_computer_security | bf6b83 | accuracy | gen | 94.00 |
| lukaemon_mmlu_high_school_microeconomics | bf6b83 | accuracy | gen | 97.06 |
| lukaemon_mmlu_professional_law | bf6b83 | accuracy | gen | 81.42 |
| lukaemon_mmlu_medical_genetics | bf6b83 | accuracy | gen | 98.00 |
| lukaemon_mmlu_professional_psychology | bf6b83 | accuracy | gen | 91.50 |
| lukaemon_mmlu_jurisprudence | bf6b83 | accuracy | gen | 89.81 |
| lukaemon_mmlu_world_religions | bf6b83 | accuracy | gen | 92.40 |
| lukaemon_mmlu_philosophy | bf6b83 | accuracy | gen | 90.68 |
| lukaemon_mmlu_virology | bf6b83 | accuracy | gen | 59.64 |
| lukaemon_mmlu_high_school_chemistry | bf6b83 | accuracy | gen | 92.61 |
| lukaemon_mmlu_public_relations | bf6b83 | accuracy | gen | 79.09 |
| lukaemon_mmlu_high_school_macroeconomics | bf6b83 | accuracy | gen | 95.13 |
| lukaemon_mmlu_human_sexuality | bf6b83 | accuracy | gen | 93.13 |
| lukaemon_mmlu_elementary_mathematics | bf6b83 | accuracy | gen | 94.97 |
| lukaemon_mmlu_high_school_physics | bf6b83 | accuracy | gen | 90.07 |
| lukaemon_mmlu_high_school_computer_science | bf6b83 | accuracy | gen | 87.00 |
| lukaemon_mmlu_high_school_european_history | bf6b83 | accuracy | gen | 89.70 |
| lukaemon_mmlu_business_ethics | bf6b83 | accuracy | gen | 86.00 |
| lukaemon_mmlu_moral_disputes | bf6b83 | accuracy | gen | 85.84 |
| lukaemon_mmlu_high_school_statistics | bf6b83 | accuracy | gen | 95.37 |
| lukaemon_mmlu_miscellaneous | bf6b83 | accuracy | gen | 96.68 |
| lukaemon_mmlu_formal_logic | bf6b83 | accuracy | gen | 96.83 |
| lukaemon_mmlu_high_school_government_and_politics | bf6b83 | accuracy | gen | 97.93 |
| lukaemon_mmlu_prehistory | bf6b83 | accuracy | gen | 93.83 |
| lukaemon_mmlu_security_studies | bf6b83 | accuracy | gen | 83.67 |
| lukaemon_mmlu_high_school_biology | bf6b83 | accuracy | gen | 94.19 |
| lukaemon_mmlu_logical_fallacies | bf6b83 | accuracy | gen | 93.25 |
| lukaemon_mmlu_high_school_world_history | bf6b83 | accuracy | gen | 92.83 |
| lukaemon_mmlu_professional_medicine | bf6b83 | accuracy | gen | 95.59 |
| lukaemon_mmlu_high_school_mathematics | bf6b83 | accuracy | gen | 93.33 |
| lukaemon_mmlu_college_medicine | bf6b83 | accuracy | gen | 88.44 |
| lukaemon_mmlu_high_school_us_history | bf6b83 | accuracy | gen | 96.08 |
| lukaemon_mmlu_sociology | bf6b83 | accuracy | gen | 94.03 |
| lukaemon_mmlu_econometrics | bf6b83 | accuracy | gen | 83.33 |
| lukaemon_mmlu_high_school_psychology | bf6b83 | accuracy | gen | 96.15 |
| lukaemon_mmlu_human_aging | bf6b83 | accuracy | gen | 82.51 |
| lukaemon_mmlu_us_foreign_policy | bf6b83 | accuracy | gen | 94.00 |
| lukaemon_mmlu_conceptual_physics | bf6b83 | accuracy | gen | 97.02 |

观察了对应输出文件夹outputs/reviews，存在答案误匹配的问题，抽取的“pred”答案不是模型最后的回答，导致结果判定为答错。
如下图，mmlu_prehistory.jsonl文件中的截图
<img width="1180" height="375" alt="Image" src="https://github.com/user-attachments/assets/db56ad7f-a377-4520-bcf5-25246e7d198f" />
<img width="1195" height="561" alt="Image" src="https://github.com/user-attachments/assets/ad86095f-f2ee-4079-9b31-f7bb82dd780e" />

感谢反馈，我看看修复一下这个问题。

模型输出目录2025/xxxx能否打包一个zip上传一下，我分析一下bad case

> 模型输出目录2025/xxxx能否打包一个zip上传一下，我分析一下bad case

上传一直failed，我传到百度网盘了，链接如下
通过网盘分享的文件：mmlu.zip
链接: https://pan.baidu.com/s/1z9BmzULiGKu8wCtL4xZ37A?pwd=mi25 提取码: mi25 
--来自百度网盘超级会员v6的分享

我看了下模型输出，主要原因是使用了默认的`few_shot_5`来回答问题，导致模型回答了多个问题的答案，评测时设置 `dataset_args={"mmlu": {"local_path": "/home/evalscope-main/data/mmlu", "few_shot_num": 0}}` 可以解决这个问题。不过这个解析问题我也看着修一下。

> 我看了下模型输出，主要原因是使用了默认的`few_shot_5`来回答问题，导致模型回答了多个问题的答案，评测时设置 `dataset_args={"mmlu": {"local_path": "/home/evalscope-main/data/mmlu", "few_shot_num": 0}}` 可以解决这个问题。不过这个解析问题我也看着修一下。

感谢回复，我试下zero shot的结果。另外不同于evalscope，在opencompass中会有多轮对话的模板，可能是这个原因导致？
```
api_meta_template = dict(
    round=[
            dict(role='HUMAN', api_role='HUMAN'),
            dict(role='BOT', api_role='BOT', generate=True),
    ],
)

#task 
...
meta_template=api_meta_template,
...
```

多轮对话的模板不影响这个，opencompass 默认是zero shot评测方法

> 多轮对话的模板不影响这个，opencompass 默认是zero shot评测方法

这是个偶发现象吗，因为我看我测试的，抽取的“pred”答案确实都是模型最后的回答

> > 多轮对话的模板不影响这个，opencompass 默认是zero shot评测方法
> 
> 这是个偶发现象吗，因为我看我测试的，抽取的“pred”答案确实都是模型最后的回答

评测规则就是根据正则表达式，抽取第一个匹配的选项，参考这里：https://github.com/modelscope/evalscope/blob/7ff9c55132bc8127e3a51b2b55e00a703ebd72ff/evalscope/metrics/completion_parsers.py#L174-L192

> > 我看了下模型输出，主要原因是使用了默认的`few_shot_5`来回答问题，导致模型回答了多个问题的答案，评测时设置 `dataset_args={"mmlu": {"local_path": "/home/evalscope-main/data/mmlu", "few_shot_num": 0}}` 可以解决这个问题。不过这个解析问题我也看着修一下。
> 
> 感谢回复，我试下zero shot的结果。另外不同于evalscope，在opencompass中会有多轮对话的模板，可能是这个原因导致？
> 
> ```
> api_meta_template = dict(
>     round=[
>             dict(role='HUMAN', api_role='HUMAN'),
>             dict(role='BOT', api_role='BOT', generate=True),
>     ],
> )
> 
> #task 
> ...
> meta_template=api_meta_template,
> ...
> ```

非常抱歉打扰了，我想问下，我的mmlu也是相差很多，请问您最后指标有说提升嘛？

@Dyw08 使用 v1.0.0 版本的evalscope再试试呢

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！