Evaluating Multimodal Generative AI with K orean Educational Standards (2025)

Abstract: This paper presents the Korean National Educational Test Benchmark (KoNET), a new benchmark designed to evaluate Multimodal Generative AI Systems using Korean national educational tests. KoNET comprises four exams: the Korean Elementary General Educational Development Test (KoEGED), Middle (KoMGED), High (KoHGED), and College Scholastic Ability Test (KoCSAT). These exams are renowned for their rigorous standards and diverse questions, facilitating a comprehensive analysis of AI performance across different educational levels. By focusing on Korean, KoNET provides insights into model performance in less-explored languages. We assess a range of models—open-source, open-access, and closed APIs—by examining difficulties, subject diversity, and human error rates. The code and dataset builder will be made fully open-source.

## introduction the advancement of large language models (llms) has spurred the integration of sophisticated generative ai systems into various applications @xcite . recent developments combining llms with computer vision have resulted in powerful multimodal llms (mllms) @xcite @xcite . however, questions remain about the true intelligence of these systems, especially their ability to generalize across novel tasks similar to human cognition. current benchmarks predominantly focus on english, overlooking the linguistic diversity worldwide and offering limited insights into low-resource languages like korean. moreover, many benchmarks do not compare ai performance to that of ˚sanghee park and geewook kim contributed equally to this work and share first authorship. to address these challenges, we introduce konet, a benchmark dataset leveraging four key korean educational tests (refer to figure 1 ). each @xcite , and scibench @xcite focus on math, computer science, and science skills. these offer a focused evaluation of ai capabilities within educational contexts. multimodal benchmarks. seedbench @xcite and mmstar @xcite provide general multimodal evaluations. notably, there are educationally focused benchmarks such as scienceqa @xcite and math-vista @xcite , which assess ai's ability with scientific and mathematical content. further, mmmu @xcite provides diverse subject evaluations, including art and medicine, while ai2d @xcite examines diagram interpretation in grade school science. korean benchmarks. korean benchmarks are limited, but efforts like k-mmlu @xcite and ko-h5 @xcite have emerged. in multimodal contexts, kvqa @xcite and cvqa @xcite focus on vqa and cultural understanding. despite the advances, there is a notable absence of korean educational benchmarks, particularly in the multimodal domain. no existing frameworks comprehensively evaluate ai's educational performance across various school subjects within a korean context. ## proposed benchmark: konet to offer a robust evaluation framework that facilitates comprehensive comparisons with human educational levels, we converts questions from korea's national educational tests into a multimodal vqa format. ## conclusion we present konet as a benchmark for evaluating multimodal generative ai models using korean educational tests. our findings reveal varying performance with multimodal inputs and highlight specific challenges. the disparity between open and closed-source models points to the need for advancements in open-source models within non-english contexts. our analysis of human error rates offers valuable insights into ai and human performance comparisons. through konet, we aim to encourage research in multimodal and multilingual ai, thereby promoting inclusivity and diversity.