Understanding LLM Development Through Longitudinal Study: Insights from the Open K o- LLM Leaderboard (2025)

Abstract: This paper conducts a longitudinal study over eleven months to address the limitations of prior research on the Open Ko-LLM Leaderboard, which have relied on empirical studies with restricted observation periods of only five months. By extending the analysis duration, we aim to provide a more comprehensive understanding of the progression in developing Korean large language models (LLMs). Our study is guided by three primary research questions: (1) What are the specific challenges in improving LLM performance across diverse tasks on the Open Ko-LLM Leaderboard over time? (2) How does model size impact task performance correlations across various benchmarks? (3) How have the patterns in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. By analyzing 1,769 models over this period, our research offers a comprehensive examination of the ongoing advancements in LLMs and the evolving nature of evaluation frameworks.

## introduction the rapid advancement of large language models (llms) @xcite has led to the creation of various leaderboards designed to evaluate their performance across a wide range of tasks @xcite @xcite @xcite . among these, the open llm leaderboard @xcite developed by hugging face @xcite has achieved significant global recognition. in the context of korean language models, the open ko-llm leaderboard @xcite was established to specifically assess llm performance within the korean language environment. while previous analyses of the open ko-llm leaderboard @xcite have provided valuable insights into llm performance, they have † corresponding author been constrained observation periods of only five months, limiting their ability to capture long-term trends. to better understand the ongoing evolution and inherent challenges in llm development, a more comprehensive and extended analysis is required. this paper addresses this gap by conducting a detailed longitudinal study of the open ko-llm leaderboard, guided by three primary research questions: first, we analyze the longitudinal changes in performance across five tasks monitored by the open ko-llm leaderboard. these tasks are designed to evaluate various capabilities of llms, including reasoning, natural language understanding, and common sense knowledge. by examining data collected over a eleven-month period, this study aims to identify which capabilities have presented the greatest challenges for llm developers, which tasks have reached performance saturation rapidly, and which tasks continue to pose significant difficulties. this analysis will provide quantitative insights into performance trends across different tasks, thereby guiding targeted research efforts and highlighting key areas that require further advancement to push the boundaries of model development. second, we explore the correlations between different tasks based on model size. this aspect of the study examines how the performance across different tasks varies depending on the scale of the model. understanding these correlations will provide insights into the interaction between model capacity and task performance, offering a deeper understanding of how scaling influences overall effectiveness across tasks. ## open ko-llm leaderboard the open ko-llm leaderboard @xcite is a pioneering platform designed to evaluate large language models (llms) specifically in the korean language, addressing the limitations of predominantly english-focused benchmarks. this leaderboard mirrors the structure of the globally recognized open llm leaderboard by hugging face @xcite , ensuring consistency and comparability across languages. it is built on two key principles: alignment with the english leaderboard and the use of private test sets to avoid data contamination, thereby enhancing evaluation robustness. the leaderboard employs the ko-h5 benchmark, comprising five tasks that assess various aspects of language understanding and generation in korean. these tasks are designed to comprehensively evaluate llm capabilities. the first task, ko-hellaswag @xcite , tests commonsense reasoning by requiring models to complete sentences contextually and logically. the second task, ko-arc @xcite , adapted from the english arc, evaluates both commonsense and scientific reasoning through multiple-choice questions. ko-mmlu @xcite , the third task, assesses multitask language understanding and domain knowledge across various subjects, requiring models to respond accurately to questions from different domains. the fourth task, ko-commongen v2 @xcite , focuses on commonsense generation, where models must create coherent sentences from given concepts, testing their ability to connect common knowledge meaningfully. lastly, ko-truthfulqa @xcite evaluates a model ability to provide truthful and accurate responses, crucial for assessing the factual integrity of llms in real-world scenarios. through the ko-h5 benchmark, the open ko-llm leaderboard provides a robust framework for evaluating korean llms and promotes linguistic diversity in llm evaluation. by incorporating tasks that reflect korean linguistic and cultural nuances, the leaderboard offers valuable insights into llm performance beyond english, encouraging a more inclusive approach to language model evaluation. ## conclusion this study provides a longitudinal analysis of the open ko-llm leaderboard, uncovering significant performance trends and underlying challenges in llm development. it was observed that smaller models consistently face scalability limitations, preventing substantial performance advancements. in contrast, larger models initially show promising improvements but eventually reach a saturation point, highlighting a critical dependency on advancements in pretrained models. these findings underscore the need for continuous innovation and enhancement in the development of pretrained models to push the boundaries of llm capabilities further. additionally, the analysis demonstrates the utility of leaderboard data in tracking the evolving dynamics of llm performance. by examining a broader range of model submissions and evaluation patterns over an extended period, this study provides valuable insights into how model size, type, and tuning methods influence overall effectiveness. such insights can inform targeted research efforts and encourage the development of strategies aimed at overcoming existing limitations, ultimately supporting more robust and adaptable llms. ogy, professor sangkeun jung from chungnam national university, and professor alice oh from kaist for their insightful advice on the open ko-llm leaderboard. finally, we deeply appreciate the open-source community for their invaluable feedback and contributions. this work was supported by institute of information & communications technology planning & evaluation(iitp) grant funded by the korea government(msit) (no. rs-2024-00338140, development of learning and utilization technology to reflect sustainability of generative language models and up-to-dateness over time).