Repository: modelscope/evalscope
Issue: llama3.1 8B base model mmlu few-shot-num 5 只有0.26
Root cause: Configuration Error

## 问题描述 /llama3.1 8B base model mmlu few-shot-num 5 只有0.26

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

## Name: evalscope
Version: 0.14.0
Summary: EvalScope: Lightweight LLMs Evaluation Framework
Home-page: https://github.com/modelscope/evalscope
Author: ModelScope team
Author-email: contact@modelscope.cn
License: 
Location: /opt/conda/lib/python3.10/site-packages
Requires: accelerate, datasets, immutabledict, jieba, jsonlines, langdetect, latex2sympy2, matplotlib, modelscope, nltk, openai, pandas, pyarrow, pyyaml, requests, rouge-chinese, rouge-score, sacrebleu, scikit-learn, seaborn, sympy, tabulate, torch, tqdm, transformers, word2number
Required-by: 




2025-04-24 10:04:17,003 - evalscope - INFO - Dump task config to ./outputs/20250424_100042/configs/task_config_236caf.yaml
2025-04-24 10:04:17,072 - evalscope - INFO - {
    "model": "xxx/Meta-llama3.1-8B-hf/Meta-Llama-3.1-8B",
    "model_id": "Meta-Llama-3.1-8B",
    "model_args": {
        "revision": "master",
        "precision": "torch.float16"
    },
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "mmlu"
    ],
    "dataset_args": {
        "mmlu": {
            "name": "mmlu",
            "dataset_id": "xxx/data-main/data/mmlu",
            "model_adapter": "multiple_choice_logits",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "high_school_european_history",
                "business_ethics",
                "clinical_knowledge",
                "medical_genetics",
                "high_school_us_history",
                "high_school_physics",
                "high_school_world_history",
                "virology",
                "high_school_microeconomics",
                "econometrics",
                "college_computer_science",
                "high_school_biology",
                "abstract_algebra",
                "professional_accounting",
                "philosophy",
                "professional_medicine",
                "nutrition",
                "global_facts",
                "machine_learning",
                "security_studies",
                "public_relations",
                "professional_psychology",
                "prehistory",
                "anatomy",
                "human_sexuality",
                "college_medicine",
                "high_school_government_and_politics",
                "college_chemistry",
                "logical_fallacies",
                "high_school_geography",
                "elementary_mathematics",
                "human_aging",
                "college_mathematics",
                "high_school_psychology",
                "formal_logic",
                "high_school_statistics",
                "international_law",
                "high_school_mathematics",
                "high_school_computer_science",
                "conceptual_physics",
                "miscellaneous",
                "high_school_chemistry",
                "marketing",
                "professional_law",
                "management",
                "college_physics",
                "jurisprudence",
                "world_religions",
                "sociology",
                "us_foreign_policy",
                "high_school_macroeconomics",
                "computer_security",
                "moral_scenarios",
                "moral_disputes",
                "electrical_engineering",
                "astronomy",
                "college_biology"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 5,
            "few_shot_random": false,
            "train_split": "train",
            "eval_split": "test",
            "prompt_template": "Answer the following multiple choice question about {subset_name}. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.\n\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "MMLU",
            "filters": null,
            "extra_params": {}
        }
    },
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_length": 2048,
        "max_new_tokens": 512,
        "do_sample": false,
        "top_k": 50,
        "top_p": 1.0,
        "temperature": 1.0
    },
    "eval_type": "checkpoint",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": null,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs/20250424_100042",
    "outputs": null,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {}
}
2025-04-24 10:04:17,074 - evalscope - INFO - **** Start evaluating on dataset /home/gomall/share/data-main/data/mmlu ****
2025-04-24 10:04:17,076 - evalscope - INFO - Loading dataset from local disk: /home/gomall/share/data-main/data/mmlu
2025-04-24 10:04:17,317 - evalscope - INFO - Use default settings: > few_shot_num: 5, > few_shot_split: train, > target_eval_split: test
2025-04-24 10:06:21,978 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_european_history.jsonl.
2025-04-24 10:06:49,894 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_business_ethics.jsonl.
2025-04-24 10:07:57,607 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_clinical_knowledge.jsonl.
2025-04-24 10:08:26,318 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_medical_genetics.jsonl.
2025-04-24 10:10:40,055 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_us_history.jsonl.
2025-04-24 10:11:19,025 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_physics.jsonl.
2025-04-24 10:13:13,381 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_world_history.jsonl.
2025-04-24 10:13:54,013 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_virology.jsonl.
2025-04-24 10:14:46,480 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_microeconomics.jsonl.
2025-04-24 10:15:22,031 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_econometrics.jsonl.
2025-04-24 10:15:55,587 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_college_computer_science.jsonl.
2025-04-24 10:17:02,714 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_biology.jsonl.
2025-04-24 10:17:34,615 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_abstract_algebra.jsonl.
2025-04-24 10:18:50,281 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_professional_accounting.jsonl.
2025-04-24 10:20:05,182 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_philosophy.jsonl.
2025-04-24 10:22:53,329 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_professional_medicine.jsonl.
2025-04-24 10:24:47,890 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_nutrition.jsonl.
2025-04-24 10:25:21,574 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_global_facts.jsonl.
2025-04-24 10:25:53,315 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_machine_learning.jsonl.
2025-04-24 10:27:28,104 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_security_studies.jsonl.
2025-04-24 10:27:59,946 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_public_relations.jsonl.
2025-04-24 10:30:20,666 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_professional_psychology.jsonl.
2025-04-24 10:32:30,462 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_prehistory.jsonl.
2025-04-24 10:33:22,791 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_anatomy.jsonl.
2025-04-24 10:34:03,588 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_human_sexuality.jsonl.
2025-04-24 10:35:08,252 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_college_medicine.jsonl.
2025-04-24 10:36:18,940 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_government_and_politics.jsonl.
2025-04-24 10:37:02,837 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_college_chemistry.jsonl.
2025-04-24 10:38:01,317 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_logical_fallacies.jsonl.
2025-04-24 10:39:05,750 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_geography.jsonl.
2025-04-24 10:41:31,135 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_elementary_mathematics.jsonl.
2025-04-24 10:42:50,090 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_human_aging.jsonl.
2025-04-24 10:43:20,809 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_college_mathematics.jsonl.
2025-04-24 10:45:07,913 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_psychology.jsonl.
2025-04-24 10:46:00,304 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_formal_logic.jsonl.
2025-04-24 10:47:05,409 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_statistics.jsonl.
2025-04-24 10:47:52,655 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_international_law.jsonl.
2025-04-24 10:48:51,768 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_mathematics.jsonl.
2025-04-24 10:49:31,031 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_computer_science.jsonl.
2025-04-24 10:50:17,295 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_conceptual_physics.jsonl.
2025-04-24 10:52:38,087 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_miscellaneous.jsonl.
2025-04-24 10:53:53,007 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_chemistry.jsonl.
2025-04-24 10:54:42,976 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_marketing.jsonl.
2025-04-24 11:08:02,089 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_professional_law.jsonl.
2025-04-24 11:09:27,252 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_management.jsonl.
2025-04-24 11:09:52,104 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_college_physics.jsonl.
2025-04-24 11:10:17,048 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_jurisprudence.jsonl.
2025-04-24 11:11:02,648 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_world_religions.jsonl.
2025-04-24 11:11:56,518 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_sociology.jsonl.
2025-04-24 11:12:30,798 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_us_foreign_policy.jsonl.
2025-04-24 11:13:54,680 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_high_school_macroeconomics.jsonl.
2025-04-24 11:14:35,124 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_computer_security.jsonl.
2025-04-24 11:20:27,763 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_moral_scenarios.jsonl.
2025-04-24 11:22:51,854 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_moral_disputes.jsonl.
2025-04-24 11:23:43,492 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_electrical_engineering.jsonl.
2025-04-24 11:24:25,447 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_astronomy.jsonl.
2025-04-24 11:25:01,354 - evalscope - INFO - Dump predictions to ./outputs/20250424_100042/predictions/Meta-Llama-3.1-8B/mmlu_college_biology.jsonl.
2025-04-24 11:25:07,447 - evalscope - INFO - Dump report: ./outputs/20250424_100042/reports/Meta-Llama-3.1-8B/mmlu.json 

2025-04-24 11:25:07,476 - evalscope - INFO - Report table: 
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Model             | Dataset   | Metric          | Subset                              |   Num |   Score | Cat.0          |
+===================+===========+=================+=====================================+=======+=========+================+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_european_history        |   165 |  0.2061 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_us_history              |   204 |  0.2843 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_world_history           |   237 |  0.2743 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | philosophy                          |   311 |  0.2444 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | prehistory                          |   324 |  0.2623 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | logical_fallacies                   |   163 |  0.2454 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | formal_logic                        |   126 |  0.3095 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | international_law                   |   121 |  0.3058 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | professional_law                    |  1534 |  0.2419 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | jurisprudence                       |   108 |  0.25   | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | world_religions                     |   171 |  0.3041 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | moral_scenarios                     |   895 |  0.2726 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | moral_disputes                      |   346 |  0.2717 | Humanities     |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | business_ethics                     |   100 |  0.29   | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | clinical_knowledge                  |   265 |  0.2377 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | medical_genetics                    |   100 |  0.36   | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | virology                            |   166 |  0.3072 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | professional_accounting             |   282 |  0.2801 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | professional_medicine               |   272 |  0.2206 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | nutrition                           |   306 |  0.2582 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | global_facts                        |   100 |  0.32   | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | anatomy                             |   135 |  0.1852 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | college_medicine                    |   173 |  0.2659 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | human_aging                         |   223 |  0.3004 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | miscellaneous                       |   783 |  0.2682 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | marketing                           |   234 |  0.2778 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | management                          |   103 |  0.2427 | Other          |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_physics                 |   151 |  0.245  | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | college_computer_science            |   100 |  0.24   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_biology                 |   310 |  0.2548 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | abstract_algebra                    |   100 |  0.18   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | machine_learning                    |   112 |  0.3214 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | college_chemistry                   |   100 |  0.24   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | elementary_mathematics              |   378 |  0.2566 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | college_mathematics                 |   100 |  0.33   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_statistics              |   216 |  0.25   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_mathematics             |   270 |  0.2296 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_computer_science        |   100 |  0.22   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | conceptual_physics                  |   235 |  0.3149 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_chemistry               |   203 |  0.1773 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | college_physics                     |   102 |  0.2843 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | computer_security                   |   100 |  0.26   | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | electrical_engineering              |   145 |  0.269  | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | astronomy                           |   152 |  0.2566 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | college_biology                     |   144 |  0.2917 | STEM           |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_microeconomics          |   238 |  0.2857 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | econometrics                        |   114 |  0.2368 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | security_studies                    |   245 |  0.2204 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | public_relations                    |   110 |  0.3273 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | professional_psychology             |   612 |  0.2533 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | human_sexuality                     |   131 |  0.2901 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_government_and_politics |   193 |  0.2176 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_geography               |   198 |  0.2576 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_psychology              |   545 |  0.2459 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | sociology                           |   201 |  0.2587 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | us_foreign_policy                   |   100 |  0.28   | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Meta-Llama-3.1-8B | mmlu      | AverageAccuracy | high_school_macroeconomics          |   390 |  0.3564 | Social Science |
+-------------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+ 

2025-04-24 11:25:07,477 - evalscope - INFO - **** Evaluation finished on /home/gomall/share/data-main/data/mmlu ****




可以用main分支代码再测试一下

请问直接pip install的evalscope不是main分支的吗？

---原始邮件---
发件人: "Yunlin ***@***.***&gt;
发送时间: 2025年4月24日(周四) 下午5:30
收件人: ***@***.***&gt;;
抄送: ***@***.******@***.***&gt;;
主题: Re: [modelscope/evalscope] llama3.1 8B base model mmlu few-shot-num 5 只有0.26 (Issue #505)


 Yunnglin left a comment (modelscope/evalscope#505)
 
可以用main分支代码再测试一下
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

@zgydbc pip install 的是打包后的版本，代码相较于main分支可能落后

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！