Open K o- LLM Leaderboard: Evaluating Large Language Models in K orean with K o-H5 Benchmark (2024)

Abstract: This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark as vital tools for evaluating Large Language Models (LLMs) in Korean. Incorporating private test sets while mirroring the English Open LLM Leaderboard, we establish a robust evaluation framework that has been well integrated in the Korean LLM community. We perform data leakage analysis that shows the benefit of private test sets along with a correlation study within the Ko-H5 benchmark and temporal analyses of the Ko-H5 score. Moreover, we present empirical support for the need to expand beyond set benchmarks. We hope the Open Ko-LLM Leaderboard sets precedent for expanding LLM evaluation to foster more linguistic diversity.

## introduction the emergence of large language models (llms) @xcite have also introduced an ever growing demand for robust evaluation frameworks for llms. while multiple benchmarks @xcite @xcite have been proposed for a more holistic evaluation of llms, they are mostly limited to the english language. recognizing the need to expand the mostly englishcentric llm benchmarks to other languages such as korean, we introduce the "open ko-llm leaderboard" and the "ko-h5 benchmark". the open ko-llm leaderboard is built on the following two principles: i) alignment with the english open llm leaderboard @xcite and ii) private test sets. enabling straightforward comparison between the two leaderboard results, following the well-established composition of the open llm leaderboard is key to the suc-cessful integration of the open ko-llm leaderboard in the korean llm community. further, our private test sets allow for robust evaluation of a plethora of models in the wild without significant worry of data contamination on the tested benchmarks @xcite @xcite . we show that our private test sets have little overlap with some of the most popular training datasets used by top models in the open ko-llm leaderboard, empirically solidifying the argument for private test sets. to reveal various key insights, we perform an extensive multi-faceted analysis. for instance, correlation between the tasks that constitute the ko-h5 benchmark shows that the newly added dataset, i.e., ko-commongen v2, differentiates the open ko-llm leaderboard from the english open llm leaderboard by bringing more diversity to the evaluation suite. additionally, analysis of the improvements in the ko-h5 score over time for differently sized models presents insights into a potential critical model size that enables rapid performance improvement. another temporal analysis of the ko-h5 benchmark scores with respect to various model types brings quantitative support for the notion that improvements in pretrained models lead to improvements in instruction-tuned models. further analysis reveals a relatively quick saturation of certain task scores, indicating the need to move beyond a set benchmark. in other words, a shift towards a more holistic evaluation scheme that better adheres to real-world use-cases is needed. building on the analytical results on score changes for each task of the top performing models, we offer a practical criteria of judging when to expand the evaluation suite for llms. our contributions can be summarized as follows: ## conclusion this paper presents the open ko-llm leaderboard and the ko-h5 benchmark as innovative tools for evaluating korean llms. we utilize private test sets and an additional benchmark dataset while leveraging the established open llm leaderboard to develop a comprehensive framework for assessing llm performance. our extensive anal-yses reveal that there is little overlap in our private test sets with some of the most popular training datasets used in the open ko-llm leaderboard submissions. further, the newly added ko-commongen v2 dataset acts as a new axis of llm evaluation, as supported by our correlation study. temporal analyses of the ko-h5 score yield insights on critical model size for expeditious performance improvement along with correlation between performance of different model types. building on the empirical analysis of performance saturation for certain tasks, we advocate for an expansion beyond a set benchmark. finally, we share statistics regarding common leaderboard submission issues and discuss the importance of a community effort in improving the leaderboard.