KMMLU : Measuring Massive Multitask Language Understanding in K orean (2025)

Abstract: We propose KMMLU, a Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean evaluation tools heavily rely on translated versions of existing English benchmarks, KMMLU is collected from original Korean exams, thereby capturing linguistic and cultural aspects of the Korean language. Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. Notably, one-fifth of the questions in KMMLU require knowledge of Korean culture for accurate resolution. KMMLU thus provides a more accurate reflection of human preferences compared to translated versions of MMLU and offers deeper insights into LLMs’ shortcomings in Korean knowledge. The dataset and codes are made publicly available for future research.

## introduction recent works often leverage translated versions of mmlu @xcite to evaluate the multilingual capabilities of large language models (llms) @xcite @xcite . however, naively translating english benchmarks into a language of interest faces critical limitations. first, machine translation can lead to a compromised dataset with issues like unnatural language, typos, and grammatical mistakes @xcite @xcite . second, mmlu, designed primarily for english speakers, includes content that assumes knowledge of the american legal system or requires familiarity with english slang and culture @xcite @xcite @xcite . thus, while translated versions hint at multilingual proficiency, they fail to capture the cultural aspects that native speakers might consider to be crucial. to address this issue for the korean nlp community, we introduce kmmlu, a comprehensive benchmark consisting of 35,030 questions spanning 45 subjects. unique to kmmlu is its sourcing: all questions are derived from korean exams, ensuring authentic korean language without any translated material. additionally, our questions are localized to korea: they reflect the cultural attitudes of koreans, rather than westerners (see figure 1 ). our comparative analysis, depicted in figure 3 , shows that kmmlu surpasses previously translated benchmarks by offering questions that are linguistically natural and steeped in the korean cultural context, providing a deeper insight into llm performance in korean (section 6). to address the shortcomings of korean benchmarking, which relies heavily on translated @xcite @xcite or private datasets @xcite @xcite , we make kmmlu and its corresponding evaluation codes publicly available 1 . we evaluate 24 different llms across 4 categories: (1) multilingual pretrained models @xcite @xcite ; (2) multilingual chat models; (3) korean pretrained models @xcite ; and (4) proprietary models including those serviced in @xcite @xcite . our results show significant room for improvement, with gpt-4 scoring the highest at 59.95%. surprisingly, we see little evidence of a "curse of multilinguality" @xcite discussed in previous work comparing bloom @xcite to monolingual english models @xcite . we conduct a detailed analysis to better understand how llms utilize korean knowledge in question-answering. initially, we observe that, despite gpt-4's overall excellence, it displays gaps in areas demanding localized knowledge, demonstrating the importance of localizing benchmarks. for example, in korean history, @xcite achieves a 35% success rate compared to hyperclova x @xcite , a koreanspecific llm, which scores 44%. notably, hyper-clova x is unique in its consistent improvement with the use of chain-of-thought (cot) prompting, indicating the challenge non-korean llms face in producing accurate korean explanations. ## conclusion in this work, we introduce the kmmlu benchmark-a comprehensive compilation of 35,030 expert-level multiple-choice questions spanning 45 subjects, all sourced from original korean exams without any translated content. our findings highlight significant room for improvement in the korean proficiency of state-of-the-art llms. we discover that the improvements in the performance of non-korean llms stem from capabilities unrelated to korean, underscoring the importance of korean pre-training for better performance in korea-specific contexts. we expect the kmmlu benchmark to aid researchers in identifying the shortcomings of current models, enabling them to assess and develop better korean llms effectively.