A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment (2025)

Abstract: Recent advancements in retrieval-augmented generation (RAG) have enhanced large language models in question answering by integrating external knowledge. However, challenges persist in achieving global understanding and aligning responses with human ethical and quality preferences. To address these issues, we propose GraphMPA, a comprehensive graph-based framework with mode-seeking preference alignment. Our approach constructs a hierarchical document graph using a general similarity measurement, mimicking human cognitive processes for information understanding and synthesis. Additionally, we introduce mode-seeking preference optimization to better align model outputs with human preferences through probability-matching constraints. Extensive experiments on six datasets demonstrate the effectiveness of our GraphMPA.

## introduction retrieval-augmented generation (rag) with large language models (llms) has recently emerged as a promising approach in question-answering (qa) @xcite . this is mainly due to its ability to retrieve external documents, thus increasing the knowledge of the model. however, despite its advances, existing studies still face the following challenges at both input and output levels. from the representation and understanding of input external documents, previous works @xcite @xcite @xcite often struggle with tasks that require comprehensive, global understanding, as they fail to provide a unified and holistic view of the relevant external knowledge. for instance, as shown in figure 1 , if we directly retrieve effective information across all input documents for an overarching query like "what about shanghai?", the potential answer may refer to sentence 6, where the pronoun "it" is interpreted as referring to "shanghai" (it actually refers to beijing.) due to the presence of the query words at the end of this sentence. only very recently, @xcite and @xcite introduced a global graph-based rag strategy to alleviate this issue, but it heavily relies on large-scale llms (e.g., gpt-4) to extract entity pairs and their corresponding relations from each sentence as figure 1 @xcite . this not only consumes significant resources but may also overlook sentences without entities or relations like the sentence 3 in figure 1 . therefore, we argue for the use of smaller, more efficient llms and the construction of sentence relevance using a more general and comprehensive measurement as figure 1 @xcite . from the quality of output answers, ensuring that llms generate responses aligned with human standards and preferences remains a significant statistical and practical challenge in qa @xcite , as it involves modeling complex patterns of human reasoning, contextual understanding, and nuanced language use. although approaches like dpo @xcite have been proposed to mitigate this issue, they lead to a compromised approximation of optimal distribution @xcite based on large-scale llms (gpt-4) generated corpus, such as a mean-seeking policy that places large mass to the mean of different modes @xcite as illustrated in figure 1(c ). however, it is insufficient since we need to estimate the exact and primary mode of the target distribution @xcite . we argue that a mode-seeking strategy based on an auto-constructed corpus with small-scale llms, can more effectively capture the alignment with human preferences as shown in figure 1(d ). to address these challenges, in this paper, we propose a comprehensive hierarchical graph framework with mode-seeking preference alignment (graphmpa) based on rag for qa. specifically, we first demolish external knowledge into a hierarchical graph using a simple but comprehensive similarity measurement, mimicking human cognitive processes in organizing and synthesizing information through abstractive summarization from low-level to high-level @xcite . next, we retrieve the top-k small documents to construct a human-preferred dataset by data synthesis for real preference alignment. finally, to achieve better mode-seeking, we introduce mode-seeking preference optimization to align the model with human preferences. in general, our contributions are summarized as follows: ## related work llms have achieved significant success in a wide range of qa tasks. however, their notable challenge is the limited access to specialized or upto-date knowledge, which can lead to outdated or incomplete responses in domain-specific tasks. to address these issues, rag has been proposed as a solution @xcite . our work is mainly related to rag and human preference alignment. retrieval-augmented generation. rag enhances llms with external retrieval, making it ideal for knowledge-intensive tasks @xcite . traditional approaches of qa @xcite normally face the "lost in the middle" problem @xcite , and are expensive and slow. rag can alleviate these issues by retrieving relevant documents from external sources and incorporating this information into the generation process. however, determining when and which documents to retrieve and global understanding issues remain challenging @xcite . therefore, @xcite propose a graph-based method for query-focused summarization by extracting an entity knowledge graph with very large-scale llms, like gpt-4. meanwhile, @xcite suggest a recursive, hierarchical summarization approach by a tree structure, but it ignores the internal relations among nodes in each layer. unlike the above studies, we design a comprehensive graph structure with simple and finegrained document relations to incorporate effective knowledge, which completely abandons large-scale llms as entity-relation extractors. human preference alignment (hpa). recently, direct preference optimization (dpo) @xcite has emerged as a promising alternative to the previously popular reinforcement learning from human feedback (rlhf) @xcite . it simplifies the training pipeline, eliminating the need for separate reward models and policy optimization @xcite . however, this often results in a mean-seeking approximation that compromises the optimal solution and relies on gpt-4 generated data for optimization. consequently, the strategy π θ tends to cover all modes of the target strategy π * , rather than concentrating on the most important modes. as a result, the generated texts or behaviors may lack clear direction or focus, failing to effectively capture the key features of human preferences. unlike previous works, to avoid the compromised mean-seeking approximation of the optimal solution in standard dpo, we introduce the modeseeking (ms) loss inspired by @xcite and design an intuitive auto-constructed dataset with small-scale llms for training. ## methodology motivation. to systematically understand the traditional fragmented pieces of knowledge, we draw inspiration from human cognitive processes in handling and organizing information, proposing to summarize closed small documents and organize them into a hierarchical graph. this also enhances both the low-level and the high-level comprehension of external knowledge. then, we simulate the human thinking process to derive the final answer through structured reasoning. this method leverages the mode-seeking preference optimization aiming to better fit the optimal pattern (mode) rather than the overall expectation. task formulation. in rag, the objective is to answer a natural language question q by querying a set of segmented documents d. each document d is typically derived from external text. the question and the retrieved documents are then passed into an llm m to generate the answer a. during the retrieval process, an embedding model embed is used to transform the text input into dense vector representations. ## experiments we conduct experiments to evaluate our method on diverse qa tasks against baselines. ## conclusion this work explores a summarization-based hierarchical graph to comprehensively extract both lowand high-level information from external knowledge for answering questions (qa). moreover, we utilize small-scale llms to automatically synthesize data on human preferences and employ modeseeking loss to capture the main patterns of the optimal policy, thereby better achieving the output of human preferences.