K o- L ong RAG : A K orean Long-Context RAG Benchmark Built with a Retrieval-Free Approach (2025)

Abstract: The rapid advancement of large language models (LLMs) significantly enhances long-context Retrieval-Augmented Generation (RAG), yet existing benchmarks focus primarily on English. This leaves low-resource languages without comprehensive evaluation frameworks, limiting their progress in retrieval-based tasks. To bridge this gap, we introduce Ko-LongRAG, the first Korean long-context RAG benchmark. Unlike conventional benchmarks that depend on external retrievers, Ko-LongRAG adopts a retrieval-free approach designed around Specialized Content Knowledge (SCK), enabling controlled and high-quality QA pair generation without the need for an extensive retrieval infrastructure. Our evaluation shows that o1 model achieves the highest performance among proprietary models, while EXAONE 3.5 leads among open-sourced models. Additionally, various findings confirm Ko-LongRAG as a reliable benchmark for assessing Korean long-context RAG capabilities and highlight its potential for advancing multilingual RAG research. The dataset and source code will be released publicly.

## introduction the rapid advancements in long-context large language models (llms) significantly enhance their ability to process and comprehend extended texts, benefiting diverse applications such as information retrieval, document summarization, and question answering @xcite . in response, numerous benchmarks are developed to evaluate the effectiveness of retrieval-augmented generation (rag) configurations @xcite . however, existing benchmarks @xcite predominantly focus on english, leaving low-resource languages without comprehensive rag evaluation frameworks @xcite . moreover, the lack of extensive knowledge bases and the scarcity of research tasks in non-english languages further complicate benchmark construction. to address this issue, we propose ko-longrag, a high-quality korean long-context rag benchmark, along with a novel approach for generating rag datasets without reliance on explicit retrieval settings. unlike conventional benchmarks that rely on existing retrievers, ko-longrag leverages a retrieval-free paradigm designed around specialized content knowledge (sck). sck refers to domain-specific knowledge that facilitates the generation of meaningful tasks without the need for an extensive retrieval infrastructure. by segmenting the corpus into domain-specific clusters and generating question-answer pairs within these clusters based on document similarity, ko-longrag effectively simulates retrieval-based scenarios while maintaining high relevance and contextual fidelity. this methodology offers several advantages: (1) it eliminates the dependency on external retrievers, ensuring applicability in low-resource settings; (2) it maintains the integrity of retrieval-like evaluation by clustering highly similar documents; and (3) it ensures the benchmark's scalability across diverse domains. moreover, ko-longrag evaluates models on answerability, which measures their ability to derive accurate answers based on the given context, reflecting the model's effectiveness in utilizing retrieved documents in a rag setting. through our experiments, we employ ko-longrag to assess the korean long-context rag performance of various llms and demonstrate the benchmark's robustness and utility. among the proprietary models, o1 @xcite demonstrates superior performance, while exaone 3.5 @xcite excels among open-sourced models. additionally, we observe a strong correlation between model size and performance, a decline in accuracy when models support shorter context lengths-highlighting the necessity of long-context processing for effective performance-and consistent robustness of results across various judge models. these findings validate ko-longrag as an effective benchmark for korean long-context rag research and underscore its potential to drive innovation in multilingual rag studies. ## related works to rigorously assess the evolving capabilities of long-context language models across dimensions such as comprehension, memory retention, and input scalability, a range of benchmarks has been proposed, including longbench @xcite , bamboo @xcite , marathon @xcite , and ruler @xcite . similarly, in the context of retrieval-augmented generation (rag), benchmarks such as ragbench @xcite and longrag @xcite have been introduced to evaluate multi-document reasoning under long-context settings. however, the majority of these benchmarks are developed for english @xcite , thereby limiting their applicability to low-resource languages. in multilingual contexts, rag evaluations typically rely on translated english datasets or multilingual llms, which inadequately capture languagespecific retrieval challenges. in particular, korean, in addition to lacking structured knowledge bases and retrieval infrastructures, also exhibits linguistic properties, such as morphological richness, agglutinative syntax, and flexible word order, that complicate token-level retrieval and semantic alignment @xcite . to address these limitations, we introduce a retrieval-free evaluation paradigm tailored to korean, enabling scalable, language-specific benchmarking of longcontext reasoning without relying on retrievers. ## ko-longrag: a korean long-context rag benchmark we propose ko-longrag, a high-quality korean long-context rag benchmark designed to enable rigorous evaluation of long-context retrievalbased reasoning in korean. we describe the benchmark creation process in section §3.1, unanswerable cases-one of the key features of ko-longrag-in section §3.2, and benchmark statistics in section §3.3. the detailed benchmark construction, including the prompts used, can be found in appendix b. ## conclusion we introduce ko-longrag, the first korean longcontext rag benchmark, addressing the lack of evaluation frameworks for non-english languages. ko-longrag employs a retrieval-free approach using sck to generate high-quality question-answer pairs. through ko-longrag, we evaluate the korean long-context rag performance of various llms, setting a new standard for korean longcontext rag evaluation.