Repository: EleutherAI/lm-evaluation-harness
Issue: Issues reproducing Llama3.2-1B results on MMLU
Root cause: Documentation Deficiency

<details>
<summary>Results Log</summary>

`hf (pretrained=/data/models/meta-llama/Llama-3.2-1B,dtype=auto,), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto (4)`
|                 Tasks                 |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu                                   |      2|none  |      |acc   |↑  |0.3107|±  |0.0039|
| - humanities                          |      2|none  |      |acc   |↑  |0.2912|±  |0.0066|
|  - formal_logic                       |      1|none  |     5|acc   |↑  |0.1984|±  |0.0357|
|  - high_school_european_history       |      1|none  |     5|acc   |↑  |0.3758|±  |0.0378|
|  - high_school_us_history             |      1|none  |     5|acc   |↑  |0.3284|±  |0.0330|
|  - high_school_world_history          |      1|none  |     5|acc   |↑  |0.3291|±  |0.0306|
|  - international_law                  |      1|none  |     5|acc   |↑  |0.4215|±  |0.0451|
|  - jurisprudence                      |      1|none  |     5|acc   |↑  |0.3889|±  |0.0471|
|  - logical_fallacies                  |      1|none  |     5|acc   |↑  |0.2761|±  |0.0351|
|  - moral_disputes                     |      1|none  |     5|acc   |↑  |0.2775|±  |0.0241|
|  - moral_scenarios                    |      1|none  |     5|acc   |↑  |0.2380|±  |0.0142|
|  - philosophy                         |      1|none  |     5|acc   |↑  |0.3183|±  |0.0265|
|  - prehistory                         |      1|none  |     5|acc   |↑  |0.3673|±  |0.0268|
|  - professional_law                   |      1|none  |     5|acc   |↑  |0.2595|±  |0.0112|
|  - world_religions                    |      1|none  |     5|acc   |↑  |0.4386|±  |0.0381|
| - other                               |      2|none  |      |acc   |↑  |0.3602|±  |0.0086|
|  - business_ethics                    |      1|none  |     5|acc   |↑  |0.3600|±  |0.0482|
|  - clinical_knowledge                 |      1|none  |     5|acc   |↑  |0.3358|±  |0.0291|
|  - college_medicine                   |      1|none  |     5|acc   |↑  |0.2890|±  |0.0346|
|  - global_facts                       |      1|none  |     5|acc   |↑  |0.2000|±  |0.0402|
|  - human_aging                        |      1|none  |     5|acc   |↑  |0.3812|±  |0.0326|
|  - management                         |      1|none  |     5|acc   |↑  |0.3301|±  |0.0466|
|  - marketing                          |      1|none  |     5|acc   |↑  |0.4103|±  |0.0322|
|  - medical_genetics                   |      1|none  |     5|acc   |↑  |0.3900|±  |0.0490|
|  - miscellaneous                      |      1|none  |     5|acc   |↑  |0.4266|±  |0.0177|
|  - nutrition                          |      1|none  |     5|acc   |↑  |0.3824|±  |0.0278|
|  - professional_accounting            |      1|none  |     5|acc   |↑  |0.2624|±  |0.0262|
|  - professional_medicine              |      1|none  |     5|acc   |↑  |0.2868|±  |0.0275|
|  - virology                           |      1|none  |     5|acc   |↑  |0.4036|±  |0.0382|
| - social sciences                     |      2|none  |      |acc   |↑  |0.3191|±  |0.0084|
|  - econometrics                       |      1|none  |     5|acc   |↑  |0.2807|±  |0.0423|
|  - high_school_geography              |      1|none  |     5|acc   |↑  |0.3687|±  |0.0344|
|  - high_school_government_and_politics|      1|none  |     5|acc   |↑  |0.3472|±  |0.0344|
|  - high_school_macroeconomics         |      1|none  |     5|acc   |↑  |0.2462|±  |0.0218|
|  - high_school_microeconomics         |      1|none  |     5|acc   |↑  |0.2605|±  |0.0285|
|  - high_school_psychology             |      1|none  |     5|acc   |↑  |0.3376|±  |0.0203|
|  - human_sexuality                    |      1|none  |     5|acc   |↑  |0.3511|±  |0.0419|
|  - professional_psychology            |      1|none  |     5|acc   |↑  |0.3023|±  |0.0186|
|  - public_relations                   |      1|none  |     5|acc   |↑  |0.3000|±  |0.0439|
|  - security_studies                   |      1|none  |     5|acc   |↑  |0.3510|±  |0.0306|
|  - sociology                          |      1|none  |     5|acc   |↑  |0.3284|±  |0.0332|
|  - us_foreign_policy                  |      1|none  |     5|acc   |↑  |0.5200|±  |0.0502|
| - stem                                |      2|none  |      |acc   |↑  |0.2829|±  |0.0080|
|  - abstract_algebra                   |      1|none  |     5|acc   |↑  |0.2600|±  |0.0441|
|  - anatomy                            |      1|none  |     5|acc   |↑  |0.3704|±  |0.0417|
|  - astronomy                          |      1|none  |     5|acc   |↑  |0.2303|±  |0.0343|
|  - college_biology                    |      1|none  |     5|acc   |↑  |0.2500|±  |0.0362|
|  - college_chemistry                  |      1|none  |     5|acc   |↑  |0.2200|±  |0.0416|
|  - college_computer_science           |      1|none  |     5|acc   |↑  |0.3000|±  |0.0461|
|  - college_mathematics                |      1|none  |     5|acc   |↑  |0.2500|±  |0.0435|
|  - college_physics                    |      1|none  |     5|acc   |↑  |0.2255|±  |0.0416|
|  - computer_security                  |      1|none  |     5|acc   |↑  |0.5000|±  |0.0503|
|  - conceptual_physics                 |      1|none  |     5|acc   |↑  |0.3447|±  |0.0311|
|  - electrical_engineering             |      1|none  |     5|acc   |↑  |0.2690|±  |0.0370|
|  - elementary_mathematics             |      1|none  |     5|acc   |↑  |0.2354|±  |0.0219|
|  - high_school_biology                |      1|none  |     5|acc   |↑  |0.3032|±  |0.0261|
|  - high_school_chemistry              |      1|none  |     5|acc   |↑  |0.2611|±  |0.0309|
|  - high_school_computer_science       |      1|none  |     5|acc   |↑  |0.3200|±  |0.0469|
|  - high_school_mathematics            |      1|none  |     5|acc   |↑  |0.2667|±  |0.0270|
|  - high_school_physics                |      1|none  |     5|acc   |↑  |0.2318|±  |0.0345|
|  - high_school_statistics             |      1|none  |     5|acc   |↑  |0.2778|±  |0.0305|
|  - machine_learning                   |      1|none  |     5|acc   |↑  |0.3571|±  |0.0455|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.3107|±  |0.0039|
| - humanities     |      2|none  |      |acc   |↑  |0.2912|±  |0.0066|
| - other          |      2|none  |      |acc   |↑  |0.3602|±  |0.0086|
| - social sciences|      2|none  |      |acc   |↑  |0.3191|±  |0.0084|
| - stem           |      2|none  |      |acc   |↑  |0.2829|±  |0.0080|
</details>

Hi, thx for the work. I'm trying to reproduce the reported results for the Llama3.2-1B model on MMLU. The result I got is 0.3107, which is lower than the 0.493 reported by [Meta](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)

Could you pls let me know if there are any specific settings I might have missed? Thx in advance!

Hi! @VoiceBeer Meta MMLU has a different prompt style, please follow this [readme](https://github.com/meta-llama/llama-recipes/tree/main/tools/benchmarks/llm_eval_harness/meta_eval) and checkout [this PR](https://github.com/meta-llama/llama-recipes/pull/801) to reproduce our MMLU number for 1B.

Thx @wukaixingxp！Appreciate it!