All That Glitters is Not Gold: Improving Robust Retrieval-Augmented Language Models with Fact-Centric Preference Alignment (2025)

Abstract: Retrieval-augmented language model (RALM) relies on retrieved external knowledge to generate responses, resulting in vulnerability in the face of retrieval results with noisy documents. Previous works integrate additional filters or finetune Large Language Models (LLMs) to learn adaptive retrieval to reduce the performance damage of noisy documents. However, prior noise filtering may lead to the loss of crucial information, and these methods do not focus on distracting documents with high semantic relevance, which is the most challenging problem. In this study, we propose a training method for fact-centric preference alignment (FPA) to improve the ability of LLMs to directly extract useful information from noisy retrieval results without prior filtering. Our method performs positive document mining based on factual consistency and uses LLMs self-generated synthetic data as training data without manual annotation. We evaluate our FPA on four question answering benchmarks, and the experimental results demonstrate that our method achieves significant improvement with a small scale of training data.

## introduction although large language models (llms) show excellent performance in many tasks such as text comprehension and reasoning @xcite @xcite , they still face challenges such as lack of knowledge and hallucinations in knowledge-intensive tasks @xcite , etc. retrieval-augmented language models (ralms) alleviate this problem by providing retrieved external information to llms @xcite @xcite . however, retrieval results cannot be perfect, and relevant information panda is a national animal of which country? china the red panda ("ailurus fulgens") is a mammal native to the eastern himalayas and southwestern china. when did big air snowboarding become an olympic sport? 2018 on july 11, 2011, the international olympic committee's executive board approved the addition of ski and snowboard slopestyle to the winter olympics roster of events, effective in 2014. …big air was added for 2018. snowboarding was one of five new sports or disciplines added to the winter olympic program between 1992 and 2002. figure 1 : examples of irrelevant documents in the retrieval results. the first example (top) shows that a document containing the ground truth may still be irrelevant. the second example (bottom) shows that two highly semantically related documents actually contain different facts. to solve this problem, previous works add a post-retrieval process to granularly select necessary information and filter out noise by re-ranking @xcite , compression @xcite @xcite , and other methods @xcite . however, these methods may suffer from loss of critical information due to excessive compression or reranking bias, leading to performance degradation. some methods directly fine-tune the llms and teach llms to adaptively decide when to retrieve, reducing the potential harm of irrelevant information by avoiding unnecessary retrieval @xcite @xcite . however, these methods do not focus on improving the ability of llms to distinguish distracting documents, and may still be misled when providing retrieved documents. in this work, we enhance the noise robustness of ralms by improving the ability of llms to distinguish between relevant and noisy documents (especially distracting documents with high semantic similarity), rather than adding additional noise filters. we expect llm to (1) refer to useful information as comprehensively as possible while avoiding referring to any irrelevant information, and (2) make use of its internal knowledge to answer the given question when the retrieval results do not contain enough useful information. to achieve this, we propose a new training method for fact-centric preference alignment called fpa. the key aspect of our method is to label useful information (i.e., positive documents) and distracting information (i.e., negative documents) from a new perspective, since they are difficult to distinguish in terms of semantic similarity. inspired by the application of the nli model in citation quality evaluation @xcite and conflict detection @xcite , we propose a fact-centric positive document mining approach that automatically labels documents based on their entailment relationship with ground truth statements, without relying on manually annotated gold documents. then, we sample responses according to positive and negative documents and design a citation mismatch data augmentation strategy to construct preference data. by fine-tuning the model for preference alignment, our approach improves llm's ability to distinguish between useful and distracting information. the main contributions of this paper are as follows: ## methodology in this section, we introduce the task formulation ( §3.1) and our approach fpa, a training method to improve the noise robustness of ralms using self-generated synthetic data from llms, as illustrated in figure 2 . our method first introduces a new dimension for mining potential positive documents ( §3.2), and samples training data for preference alignment based on positive and negative documents. we then use dpo to improve the model's ability to focus on relevant information while excluding distracting context ( §3.3) and integrate adaptive retrieval prompting during inference ( §3.4). ## conclusion in this work, we propose fpa, a new fact-centric preference alignment training method for improving the noise robustness of ralms. fpa first utilizes the entailment relationship between documents and ground truth statements for positive document mining, and constructs preference data by sampling llm responses according to positive and negative documents. the preference data are used to align llm for enhancing its capacity to distinguish distracting documents and generate correct answers. experimental results on four qa benchmarks indicate that our approach achieves higher accuracy than existing methods with less training data, and exhibits strong generalization from short-form qa to long-form qa.