From KMMLU -Redux to Pro: A Professional K orean Benchmark Suite for LLM Evaluation (2025)

Abstract: The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applicability in real-world scenarios. In this paper, we introduce two Korean expert-level benchmarks. KMMLU-Redux, reconstructed from the existing KMMLU consists of questions from the Korean National Technical Qualification exams, with critical errors removed to enhance reliability. KMMLU-Pro is based on Korean National Professional Licensure exams to reflect professional knowledge in Korea. Our experiments demonstrate that these benchmarks comprehensively represent industrial knowledge in Korea.

## introduction as llms continue to achieve strong performance across a wide range of subjects @xcite @xcite , the demand for comprehensive benchmarks has grown. mmlu @xcite is widely used for its broad coverage of general knowledge from elementary to college level. however, its publicly available online problems has raised concerns about reliability and potential data contamination @xcite @xcite . we identify similar issues in kmmlu @xcite , a widely used benchmark for evaluating korean expert-level knowledge. the dataset was constructed by crawling websites that provide questions from various exams. we observe noisy samples, including problems that explicitly reveal the answer or non-existent reference, which can mislead performance evaluation. additionally, we find evidence of contamination between the train and test splits, as well as with common web corpus such as fineweb2 @xcite . instead of collecting data from online sources directly, recent challenging benchmarks @xcite @xcite @xcite have been constructed through problems and answers authored by human expert. although this approach ensures high-quality, contamination-free benchmarks, it is costly to construct and maintain. the high construction cost hinders regular updates @xcite , leaving the benchmarks vulnerable to deprecation and contamination. furthermore, existing benchmarks primarily focus on academic knowledge and often overlook the practical applicability of models in industrial or professional contexts. as llms are increasingly adopted in industrial domains @xcite , it becomes essential to assess whether they possess the necessary expertise to support tasks that require certified knowledge. for example, before deploying an llm as a legal assistant, one must ensure that the model can reliably understand to meet professional certification standards. we introduce two benchmarks to address the limitations above and incorporate professional knowledge to evaluate the practical applicability of llms. first, kmmlu-redux, a refined subset of kmmlu with 2,587 problems, is built through manual examination by authors to reduce errors and contamination within kmmlu. while the kmmlu contains wide range of problems, even in high-school level, kmmlu-redux only selects korean national technical qualification (kntq) exams as sources. the exams require applicants to have either a bachelor's degree or at least nine 19067 years experience in industrial field, thus making the benchmark more challenging. second, we build kmmlu-pro, a new challenging benchmark, which consists of 2,822 problems from acquisition exams for korean national professional licensure (knpl), representing highly specialized professions in korea. we include 14 professions ( we conduct extensive evaluations of various llms on the two benchmarks. our benchmarks, based on real-world exams, enable aligned analysis with industrial and professional qualifications, effectively revealing the practical strengths of each model. as shown in figure 1 moreover, we observe the significant performance gaps between our datasets and merely translated datasets like mmmlu (openai, 2024), especially in domains such as laws, where in-depth knowledge of specific countries is required (see section 6.1). we argue that these findings underscore the practicality of our benchmarks for assessing the capabilities of models in professional fields within korea. to summarize, our contributions are as follows: ## kmmlu-redux we first revisit kmmlu @xcite to examine its quality. building on these insights, we construct kmmlu-redux, a cleaned and compact version of the kmmlu. we carefully denoise against the kmmlu and increase difficulty. ## kmmlu-pro a major challenge in building benchmarks from online sources is data contamination @xcite @xcite . while some studies address this by having experts manually create problems @xcite @xcite @xcite , this approach is costly and time-consuming. as an alternative, recent work explores periodic releases of fresh subsets. @xcite . motivated by these approaches, we focus on the korean national professional licensure (knpl) exams. these are high-stakes exams administered annually that pose a significant challenge. unlike benchmarks crafted by a small set of experts @xcite , our approach leverages the well-established curricula of professional licensing systems, designed to assess real-world professional knowledge. ## experiments we select a diverse set of baseline models varying in size, multilingual capability, and reasoning ability. by default, we apply a zero-shot chainof-thought (cot) @xcite prompt written in korean. however, in early experiments, we observed that some reasoning models perform worse when prompted in korean. therefore, we evaluate those models using both korean and english prompts and report the scores with the highest average score across kmmlu-redux and kmmlu-pro foot_3 . we use greedy decoding for nonreasoning models, while for reasoning models, we adopt the temperature of 0.6 and top-p (holtzman et al., 2020) of 0.95. for more details, see appendix e. evaluating open-weight models was conducted over four days using sixteen nvidia a100 gpus. for closed models, we accessed them via api calls, which incurred a total cost of approximately $4,000. metrics in addition to accuracy, our primary evaluation metric, we also report the number of licensure exams each llm passes in kmmlu-pro. to better align with human evaluation standards, our procedure is designed to mirror the official certification standards. however, for licenses that rely heavily on image-based questions, full replication is not possible. see appendix e.3 for details. ## conclusion we present two benchmarks constructed from realworld professional licensing exams, designed to reflect industrial domain knowledge and practical application standards. to ensure reliability, we identify and eliminate various sources of errors. through extensive experiments, we evaluate the professional knowledge capabilities of llms across a wide range of domains. our analysis further identifies key factors that influence performance, including region-specific knowledge, reasoning budget, and prompt language. we hope this work provides a foundation for more rigorous evaluation and continued advancement of real-world competence in language models.