Tree-of-Question: Structured Retrieval Framework for K orean Question Answering Systems (2024)

Abstract: We introduce Korean language-specific RAG-based QA systems, primarily through the innovative Tree-of-Question (ToQ) methodology and enhanced query generation techniques. We address the complex, multi-hop nature of real-world questions by effectively integrating advanced LLMs with nuanced query planning. Our comprehensive evaluations, including a newly created Korean multi-hop QA dataset, demonstrate our method’s ability to elevate response validity and accuracy, especially in deeper levels of reasoning. This paper not only showcases significant progress in handling the intricacies of Korean linguistic structures but also sets a new standard in the development of context-aware and linguistically sophisticated QA systems.

## introduction recent advancements in large language models (llms) have revolutionized information seeking and text generation, with real-world applications such as bing chat 1 , perplexity.ai foot_0 , and google bard foot_1 leading the way. these systems utilize the retrieval augmented generation (rag) methodology, where responses are crafted based on information extracted from retrieved documents. this approach allows these platforms to provide answers that not only are contextually relevant but also cite the sources they reference, enhancing the reliability and transparency of the information provided. this strategy is particularly effective in addressing the inherent issue of hallucinations in llms, where llms might generate plausible but factually incorrect information. meanwhile, a system named "naver cue:" foot_2 @xcite has emerged as a noteworthy addition to the realm of question and answering systems, specifically tailored to the korean language. it is a system grounded in hyperclovax foot_3 @xcite @xcite , a korean large language model with billions of parameters. utilizing the rag methodology and leveraging naver's search engine, "naver cue:" efficiently provides contextually relevant and accurate responses. the integration of hyperclovax into this system enables a sophisticated understanding and processing of korean language nuances, ensuring precise and relevant answers. this makes "naver cue:" a notable advancement in the domain of llm-based qa systems tailored for korean language. in the realm of rag-based qa systems powered by llms, the operational mechanism typically unfolds in a structured three-step process. initially, the system generates a query derived from the user's question, effectively translating the user's inquiry into a format suitable for search engines. following this, the query is processed through the search engine, which conducts a comprehensive search to gather relevant information. the final step involves generating a response by synthesizing and summarizing the search results into a coherent answer. a critical aspect of this process is the creation of queries that are specifically optimized for each search engine. this optimization is crucial as it directly influences the relevance and accuracy of the information retrieved. furthermore, the nature of questions posed by users in such systems is predominantly multi-hop. these multi-hop questions are multifaceted, often characterized by their inherent ambiguities or the need to collate information from multiple sources @xcite @xcite . this complexity poses additional challenges in query formulation, making it imperative for the rag-based qa systems to have advanced understanding and processing capabilities. particularly in the case of the korean language, accurately interpreting the user's question becomes more challenging due to the duality and connotation of words @xcite . korean often involves subtle nuances and implied meanings that can significantly alter the context of a query. additionally, korean is an agglutinative language, characterized by its unique grammatical structure where particles follow nouns, and the stems of verbs or adjectives are followed by endings. these endings express various grammatical properties, adding layers of complexity to the language @xcite @xcite . these linguistic features can lead to difficulties in generating search queries that precisely mirror the user's intent. therefore, rag-based qa systems designed for korean must possess enhanced capabilities to discern and reflect these subtleties. recent research in closed-book qa systems utilizes metrics like rouge-l and disambig-f1 for performance evaluation, comparing model predictions with ground truth answers @xcite . these metrics assess end-to-end performance, highlighting how closely model responses match expected answers. however, they fall short in evaluating specific aspects crucial to llm-based qa systems such as query generation, document retrieval, and response generation. in real-world scenarios, where correct answers aren't predetermined, this becomes a challenge. to address these limitations, new metrics like citation recall/precision @xcite and factscore @xcite have been introduced. these focus on evaluating the system's ability to reference relevant documents and the relevance of summarized responses to posed questions. despite these advancements, a notable gap remains: there is currently no established metric for evaluating the appropriateness of the queries generated by the in-house search engine in response to the user's questions. this highlights a crucial area for further research and development, as the ability to generate accurate and relevant queries is fundamental to the success of real-world qa systems. to effectively address the challenges in the current landscape of llm-based open-domain qa systems, particularly for the korean language, our research introduces several pivotal contributions, summarized as follows: ## task definition in llm-based retrieval augmented generation (rag) qa systems, we define the process as a sequence of functions, each transforming an input to produce an output that serves as the input for the subsequent function. let q be the user's question. then, the process can be formalized using the following notation: 1. query generation: a function f qg takes q and generates a query q. this involves interpreting q and restructuring it into a format optimized for the in-house search engine. these documents are relevant to the query and contain information pertinent to answering q. ## enhanced query planning with tree-of-question and query evaluator in the korean rag-based long-form qa system, complex questions often require structuring into simpler, searchable queries. previous studies have explored various aspects of multi-hop reasoning in closed-book qa systems. in the study by @xcite , question types requiring multi-hop reasoning were classified into three primary categories: bridging, intersection, and comparison. furthermore, the research conducted by @xcite , which draws inspiration from reasoning chains in llms. this approach entails formulating explanations as a sequence of interconnects. additionally, @xcite ; @xcite proposed methods of generating questions sequentially and creating follow-up questions when necessary. this sequential approach is valuable for developing a deeper understanding of the topic in question and ensuring comprehensive coverage. however, these methods encounter limitations in responding to questions that search for multiple queries in parallel and then synthesize these answers. such complex scenarios, referred to as the "hybrid" type in ## conclusion in this paper, we introduce advancements in ragbased qa systems for korean, focusing on the treeof-question (toq) methodology and enhanced query planning. our evaluations show the toq method's effectiveness in multi-hop reasoning and its adaptability across a comprehensive dataset. notably, toq significantly improves handling complex korean language queries by enabling deeper reasoning. additionally, we present a novel evaluation method in a detailed korean multi-hop qa dataset. our contributions pave the way for more accurate and context-sensitive qa systems, especially for languages with unique challenges like korean.