Repository: EleutherAI/lm-evaluation-harness
Issue: refactor `kmmlu` task; aggregate results by group
Root cause: Unimplemented Feature Gap

Current implementation for `kmmlu` task does not provide aggregation for subjects and categories as well as total averaging score.
I want to make this as mmlu style. (make group for each category and subject)

Current result table (do not provide any aggregations): 

|                         Tasks                         |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|-------------------------------------------------------|------:|------|-----:|-----------|---|-----:|---|-----:|
|kmmlu_direct_accounting                                |      2|none  |     5|exact_match|↑  |0.5000|±  |0.0503|
|kmmlu_direct_agricultural_sciences                     |      2|none  |     5|exact_match|↑  |0.4040|±  |0.0155|
|kmmlu_direct_aviation_engineering_and_maintenance      |      2|none  |     5|exact_match|↑  |0.5090|±  |0.0158|
|kmmlu_direct_biology                                   |      2|none  |     5|exact_match|↑  |0.4120|±  |0.0156|
|kmmlu_direct_chemical_engineering                      |      2|none  |     5|exact_match|↑  |0.5330|±  |0.0158|
|kmmlu_direct_chemistry                                 |      2|none  |     5|exact_match|↑  |0.5367|±  |0.0204|
|kmmlu_direct_civil_engineering                         |      2|none  |     5|exact_match|↑  |0.4900|±  |0.0158|
|kmmlu_direct_computer_science                          |      2|none  |     5|exact_match|↑  |0.7830|±  |0.0130|
|kmmlu_direct_construction                              |      2|none  |     5|exact_match|↑  |0.3990|±  |0.0155|
|kmmlu_direct_criminal_law                              |      2|none  |     5|exact_match|↑  |0.4000|±  |0.0347|
|kmmlu_direct_ecology                                   |      2|none  |     5|exact_match|↑  |0.5370|±  |0.0158|
|kmmlu_direct_economics                                 |      2|none  |     5|exact_match|↑  |0.5538|±  |0.0438|
|kmmlu_direct_education                                 |      2|none  |     5|exact_match|↑  |0.7300|±  |0.0446|
|kmmlu_direct_electrical_engineering                    |      2|none  |     5|exact_match|↑  |0.3720|±  |0.0153|
|kmmlu_direct_electronics_engineering                   |      2|none  |     5|exact_match|↑  |0.6080|±  |0.0154|
|kmmlu_direct_energy_management                         |      2|none  |     5|exact_match|↑  |0.3810|±  |0.0154|
|kmmlu_direct_environmental_science                     |      2|none  |     5|exact_match|↑  |0.4090|±  |0.0156|
|kmmlu_direct_fashion                                   |      2|none  |     5|exact_match|↑  |0.5170|±  |0.0158|
|kmmlu_direct_food_processing                           |      2|none  |     5|exact_match|↑  |0.4610|±  |0.0158|
|kmmlu_direct_gas_technology_and_engineering            |      2|none  |     5|exact_match|↑  |0.4160|±  |0.0156|
|kmmlu_direct_geomatics                                 |      2|none  |     5|exact_match|↑  |0.4560|±  |0.0158|
|kmmlu_direct_health                                    |      2|none  |     5|exact_match|↑  |0.6800|±  |0.0469|
|kmmlu_direct_industrial_engineer                       |      2|none  |     5|exact_match|↑  |0.4900|±  |0.0158|
|kmmlu_direct_information_technology                    |      2|none  |     5|exact_match|↑  |0.7520|±  |0.0137|
|kmmlu_direct_interior_architecture_and_design          |      2|none  |     5|exact_match|↑  |0.6010|±  |0.0155|
|kmmlu_direct_korean_history                            |      2|none  |     5|exact_match|↑  |0.3600|±  |0.0482|
|kmmlu_direct_law                                       |      2|none  |     5|exact_match|↑  |0.5250|±  |0.0158|
|kmmlu_direct_machine_design_and_manufacturing          |      2|none  |     5|exact_match|↑  |0.5420|±  |0.0158|
|kmmlu_direct_management                                |      2|none  |     5|exact_match|↑  |0.6250|±  |0.0153|
|kmmlu_direct_maritime_engineering                      |      2|none  |     5|exact_match|↑  |0.5050|±  |0.0204|
|kmmlu_direct_marketing                                 |      2|none  |     5|exact_match|↑  |0.8250|±  |0.0120|
|kmmlu_direct_materials_engineering                     |      2|none  |     5|exact_match|↑  |0.5270|±  |0.0158|
|kmmlu_direct_math                                      |      2|none  |     5|exact_match|↑  |0.3400|±  |0.0274|
|kmmlu_direct_mechanical_engineering                    |      2|none  |     5|exact_match|↑  |0.4590|±  |0.0158|
|kmmlu_direct_nondestructive_testing                    |      2|none  |     5|exact_match|↑  |0.5430|±  |0.0158|
|kmmlu_direct_patent                                    |      2|none  |     5|exact_match|↑  |0.3700|±  |0.0485|
|kmmlu_direct_political_science_and_sociology           |      2|none  |     5|exact_match|↑  |0.6400|±  |0.0278|
|kmmlu_direct_psychology                                |      2|none  |     5|exact_match|↑  |0.5130|±  |0.0158|
|kmmlu_direct_public_safety                             |      2|none  |     5|exact_match|↑  |0.4270|±  |0.0156|
|kmmlu_direct_railway_and_automotive_engineering        |      2|none  |     5|exact_match|↑  |0.4200|±  |0.0156|
|kmmlu_direct_real_estate                               |      2|none  |     5|exact_match|↑  |0.5000|±  |0.0354|
|kmmlu_direct_refrigerating_machinery                   |      2|none  |     5|exact_match|↑  |0.4170|±  |0.0156|
|kmmlu_direct_social_welfare                            |      2|none  |     5|exact_match|↑  |0.6660|±  |0.0149|
|kmmlu_direct_taxation                                  |      2|none  |     5|exact_match|↑  |0.4300|±  |0.0351|
|kmmlu_direct_telecommunications_and_wireless_technology|      2|none  |     5|exact_match|↑  |0.6340|±  |0.0152|

mmlu result table:
|                 Tasks                 |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu                                   |      2|none  |      |acc   |↑  |0.7421|±  |0.0035|
| - humanities                          |      2|none  |      |acc   |↑  |0.6801|±  |0.0065|
|  - formal_logic                       |      1|none  |     5|acc   |↑  |0.5635|±  |0.0444|
|  - high_school_european_history       |      1|none  |     5|acc   |↑  |0.8424|±  |0.0285|
|  - high_school_us_history             |      1|none  |     5|acc   |↑  |0.8725|±  |0.0234|
|  - high_school_world_history          |      1|none  |     5|acc   |↑  |0.8734|±  |0.0216|
|  - international_law                  |      1|none  |     5|acc   |↑  |0.8347|±  |0.0339|
|  - jurisprudence                      |      1|none  |     5|acc   |↑  |0.8426|±  |0.0352|
|  - logical_fallacies                  |      1|none  |     5|acc   |↑  |0.8160|±  |0.0304|
|  - moral_disputes                     |      1|none  |     5|acc   |↑  |0.7775|±  |0.0224|
|  - moral_scenarios                    |      1|none  |     5|acc   |↑  |0.5799|±  |0.0165|
|  - philosophy                         |      1|none  |     5|acc   |↑  |0.7974|±  |0.0228|
|  - prehistory                         |      1|none  |     5|acc   |↑  |0.8611|±  |0.0192|
|  - professional_law                   |      1|none  |     5|acc   |↑  |0.5326|±  |0.0127|
|  - world_religions                    |      1|none  |     5|acc   |↑  |0.8655|±  |0.0262|
| - other                               |      2|none  |      |acc   |↑  |0.7702|±  |0.0073|
|  - business_ethics                    |      1|none  |     5|acc   |↑  |0.7500|±  |0.0435|
|  - clinical_knowledge                 |      1|none  |     5|acc   |↑  |0.7736|±  |0.0258|
|  - college_medicine                   |      1|none  |     5|acc   |↑  |0.7168|±  |0.0344|
|  - global_facts                       |      1|none  |     5|acc   |↑  |0.5000|±  |0.0503|
|  - human_aging                        |      1|none  |     5|acc   |↑  |0.7399|±  |0.0294|
|  - management                         |      1|none  |     5|acc   |↑  |0.9320|±  |0.0249|
|  - marketing                          |      1|none  |     5|acc   |↑  |0.9359|±  |0.0160|
|  - medical_genetics                   |      1|none  |     5|acc   |↑  |0.8300|±  |0.0378|
|  - miscellaneous                      |      1|none  |     5|acc   |↑  |0.8582|±  |0.0125|
|  - nutrition                          |      1|none  |     5|acc   |↑  |0.7843|±  |0.0236|
|  - professional_accounting            |      1|none  |     5|acc   |↑  |0.5426|±  |0.0297|
|  - professional_medicine              |      1|none  |     5|acc   |↑  |0.7904|±  |0.0247|
|  - virology                           |      1|none  |     5|acc   |↑  |0.5783|±  |0.0384|
| - social sciences                     |      2|none  |      |acc   |↑  |0.8378|±  |0.0065|
|  - econometrics                       |      1|none  |     5|acc   |↑  |0.6579|±  |0.0446|
|  - high_school_geography              |      1|none  |     5|acc   |↑  |0.8788|±  |0.0233|
|  - high_school_government_and_politics|      1|none  |     5|acc   |↑  |0.9378|±  |0.0174|
|  - high_school_macroeconomics         |      1|none  |     5|acc   |↑  |0.7872|±  |0.0208|
|  - high_school_microeconomics         |      1|none  |     5|acc   |↑  |0.8908|±  |0.0203|
|  - high_school_psychology             |      1|none  |     5|acc   |↑  |0.9101|±  |0.0123|
|  - human_sexuality                    |      1|none  |     5|acc   |↑  |0.8092|±  |0.0345|
|  - professional_psychology            |      1|none  |     5|acc   |↑  |0.7990|±  |0.0162|
|  - public_relations                   |      1|none  |     5|acc   |↑  |0.7000|±  |0.0439|
|  - security_studies                   |      1|none  |     5|acc   |↑  |0.8082|±  |0.0252|
|  - sociology                          |      1|none  |     5|acc   |↑  |0.8557|±  |0.0248|
|  - us_foreign_policy                  |      1|none  |     5|acc   |↑  |0.9100|±  |0.0288|
| - stem                                |      2|none  |      |acc   |↑  |0.7136|±  |0.0078|
|  - abstract_algebra                   |      1|none  |     5|acc   |↑  |0.5800|±  |0.0496|
|  - anatomy                            |      1|none  |     5|acc   |↑  |0.7111|±  |0.0392|
|  - astronomy                          |      1|none  |     5|acc   |↑  |0.8553|±  |0.0286|
|  - college_biology                    |      1|none  |     5|acc   |↑  |0.8611|±  |0.0289|
|  - college_chemistry                  |      1|none  |     5|acc   |↑  |0.5700|±  |0.0498|
|  - college_computer_science           |      1|none  |     5|acc   |↑  |0.7000|±  |0.0461|
|  - college_mathematics                |      1|none  |     5|acc   |↑  |0.4900|±  |0.0502|
|  - college_physics                    |      1|none  |     5|acc   |↑  |0.5588|±  |0.0494|
|  - computer_security                  |      1|none  |     5|acc   |↑  |0.8300|±  |0.0378|
|  - conceptual_physics                 |      1|none  |     5|acc   |↑  |0.7787|±  |0.0271|
|  - electrical_engineering             |      1|none  |     5|acc   |↑  |0.7586|±  |0.0357|
|  - elementary_mathematics             |      1|none  |     5|acc   |↑  |0.7407|±  |0.0226|
|  - high_school_biology                |      1|none  |     5|acc   |↑  |0.8742|±  |0.0189|
|  - high_school_chemistry              |      1|none  |     5|acc   |↑  |0.6847|±  |0.0327|
|  - high_school_computer_science       |      1|none  |     5|acc   |↑  |0.8600|±  |0.0349|
|  - high_school_mathematics            |      1|none  |     5|acc   |↑  |0.5667|±  |0.0302|
|  - high_school_physics                |      1|none  |     5|acc   |↑  |0.5960|±  |0.0401|
|  - high_school_statistics             |      1|none  |     5|acc   |↑  |0.6944|±  |0.0314|
|  - machine_learning                   |      1|none  |     5|acc   |↑  |0.5714|±  |0.0470|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.7421|±  |0.0035|
| - humanities     |      2|none  |      |acc   |↑  |0.6801|±  |0.0065|
| - other          |      2|none  |      |acc   |↑  |0.7702|±  |0.0073|
| - social sciences|      2|none  |      |acc   |↑  |0.8378|±  |0.0065|
| - stem           |      2|none  |      |acc   |↑  |0.7136|±  |0.0078|