P areto RAG : Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation (2025)

Abstract: While Retrieval-Augmented Generation systems enhance Large Language Models by incorporating external knowledge, they still face persistent challenges in retrieval inefficiency and the inability of LLMs to filter out irrelevant information. We presentParetoRAG, an unsupervised framework that optimizes RAG systems through sentence-level refinement guided by the Pareto principle. By decomposing paragraphs into sentences and dynamically re-weighting core content while preserving contextual coherence, ParetoRAG achieves dual improvements in retrieval precision and generation quality without requiring additional training or API resources, while using only 40% of the tokens compared to traditional RAG approaches. This framework has been empirically validated across various datasets, LLMs, and retrievers. Furthermore, we show that ParetoRAG’s architectural improvements are orthogonally compatible with adaptive noise-robust models, enabling retrieval-augmented optimization and robust training to enhance generation quality mutually. This highlights complementary architectural refinements and noise mitigation, offering insights for integrating retrieval augmentation with robustness enhancement.

## introduction with the development of large language models (llms), their general capabilities have become increasingly powerful @xcite . however, even the most advanced llms still face challenges with factual errors @xcite . one major limitation lies in their static parametric memory, which prevents them from adapting to dynamically evolving knowledge demands or covering unknown domains beyond their training data figure 1 : the examples show that much noise impedes the llm from acquiring accurate knowledge from the retrieved content and could potentially misdirect its reasoning. finding the correct answer relies on the ability of llm to identify a small portion of key information. @xcite . llms are prone to generating plausible hallucinations but lack factual accuracy @xcite . these challenges significantly hinder the performance of llms in knowledgeintensive tasks @xcite . to address these limitations, retrieval-augmented generation (rag) @xcite @xcite integrates relevant passages from external databases into the input context, effectively enhancing the reliability and performance of models in open-domain question answering and dynamic knowledge retrieval tasks. however, the effectiveness of rag highly depends on the quality of the retrieved information @xcite . additionally, interference from redundant information and increased input length are critical factors that significantly impact model performance. in the retrieval stage, the relevance scores of core sentences can be overshadowed by redundant content at the same passage level, reducing the prominence of key information in the retrieved content. in the generation stage, retrievin contrast, our method utilizes a preprocessed sentence-level corpus, assigning higher weights to key sentences while appropriately preserving and weighting contextual information to avoid loss of coherence. inspired by the pareto principle (the 80/20 rule), this design emphasizes critical information while maintaining necessary semantic consistency. the selected sentences are fed into the llm, resulting in more accurate answers. retrieval-augmented language models (ralms) @xcite , long-context llms @xcite , and adaptive noise-robust model @xcite can be considered as solutions. these models enhance the ability to process longform text and improve the robustness of noisy information, enabling them to focus more effectively on key information and reduce the impact of redundant content. however, these approaches require additional training resources and high computational costs for further training and model finetuning. another possible solution is to reduce the granularity of retrieval from the document level to the sentence level @xcite . however, this approach may inadvertently lose some important contextual information (e.g., in the example in figure 1 , "the season" refers to chicago fire season 4 in one context and season 2 in another), which is crucial for accurately answering the given query @xcite . therefore, we propose a method that does not require additional training resources while effectively preserving contextual information and reducing document redundancy. in this work, we present paretorag, an unsupervised framework built upon the rag system. our approach leverages a preprocessed sentencelevel corpus, assigning higher weights to key sentences while carefully preserving and weighting contextual information to maintain coherence. drawing inspiration from the pareto principle (the 80/20 rule), paretorag prioritizes critical information while ensuring semantic consistency, effectively enhancing both the retrieval and generation stages of the rag pipeline. notably, paretorag requires neither additional training resources nor extra api calls. the overall paretorag framework is illustrated in figure 2 . we validate paretorag across three datasets over three retrievers. our main contributions are as follows: retrieval-augmented generation with noisy context rag @xcite is considered a useful method to address hallucinations, which improves the input questions of generative llm with retrieved documents. it usually provides an extra knowledge source from a specific corpus, i.e., wikipedia, which greatly improves the performance of llm in a variety of tasks, especially in the knowledge-intensive ones @xcite . however, due to the limitation of retrieval capabilities, retrieval-augmented systems inevitably introduce irrelevant or partially relevant knowledge to the models @xcite . in recent years, the impact of noisy information on the performance of rag systems has received increasing attention @xcite . some studies @xcite have shown that the introduction of irrelevant noise significantly degrades model performance. further analyses @xcite indicate that as the proportion of noise in the retrieval context increases, the performance of large language models (llms) deteriorates significantly. in addition, research @xcite has explored the effects of different types of noise on rag systems and found that counterfactual retrieval noise has the most detrimental impact on retrieval systems. advanced rag many advanced approaches have been developed from the original rag in recent years @xcite @xcite . considering that retrieval is sometimes unnecessary for some queries, responses without retrieval are even more accurate in many situations. selfrag @xcite is proposed to selectively retrieve knowledge and introduce a critical model to decide whether to retrieve it. sail @xcite is tuned on instructions to insert the retrieved documents before the instructions. recomp @xcite is designed to refine the retrieved passages by either abstractively or extractively summarizing them with additional models. compared with recent studies @xcite that are the most relevant to our work, a primary difference should be highlighted. dense x reduces information redundancy by degrading documents into proposition sentences. the authors trained a fine-tuned text generation model to decompose paragraphs into propositions to supplement contextual information. in contrast, our approach utilizes the sentence-context weighted attention mechanism to supplement contextual information without requiring any fine-tuning. ## method in this section, we introduce a novel framework, paretorag, for improving the accuracy of retrieval results by leveraging sentence-level weighting inspired by the pareto principle integrated into the rag system. notably, paretorag requires neither additional training resources nor extra api calls. ## experiment setups in this section, we describe the experimental setup for evaluating paretorag across various scenarios. the specific model parameters can be found in appendix e. the selection and meaning of the evaluation metrics can be found in appendix f. ## experimental results and analysis in this section, we show the overall experimental results with in-depth analyses of our framework. we also provide a cost analysis in the appendix i. ## conclusion in this work, we propose paretorag, an unsupervised framework that enhances rag systems through sentence-level optimization guided by the pareto principle. by decomposing paragraphs into sentences and dynamically re-weighting critical content while preserving contextual coherence, paretorag achieves dual improvements in retrieval precision and generation quality without requiring additional training or api resources. extensive experiments demonstrate its effectiveness: the framework reduces token consumption by 70% while improving the accuracy and fluency of the answers in diverse datasets, llms, and retrievers. our analysis further reveals synergistic effects when integrating paretorag with robustly trained language models, suggesting enhanced generalization capabilities. this study not only validates the viability of resource-efficient sentence-level refinement for rag systems but also opens avenues for exploring hybrid methodologies that com-bine retrieval-augmented mechanisms with adaptive training strategies.