R ose RAG : Robust Retrieval-augmented Generation with Small-scale LLM s via Margin-aware Preference Optimization (2025)

Abstract: Large language models (LLMs) have achieved impressive performance but face high computational costs and latency, limiting their deployment in resource-constrained settings. In contrast, small-scale LLMs (SLMs) are more efficient yet struggle to capture evolving real-world knowledge. Retrieval-augmented generation (RAG) helps by integrating external knowledge, but imperfect retrieval can introduce distracting noise that misleads SLMs. We propose RoseRAG, a robust RAG framework for SLMs via Margin-aware Preference Optimization. RoseRAG employs multi-turn prompting for detailed reasoning, rejection sampling for high-quality explanations, and contrastive preference selection to refine responses by maximizing the likelihood gap between preferred and non-preferred outputs. By integrating these components into a margin-aware optimization process, RoseRAG robustly enhances the accuracy and reliability of SLMs for RAG applications. Extensive experiments on three open-domain question answering benchmarks indicate that our innovative RoseRAG surpasses state-of-the-art baselines significantly.

## introduction large language models (llms) have demonstrated remarkable capabilities in a wide array of natural language processing tasks @xcite @xcite . however, these powerful models are typically large-scale, requiring substantial computational resources for training, and often incurring high latency during inference @xcite . such limitations serve as a key hurdle that prevents these models from being deployed in real practice. in contrast, small-scale llms (slms) offer a viable alternative by providing high utility while remaining computationally efficient and easier to deploy in resource-constrained environments @xcite . despite their efficiency, slms are fundamentally constrained by their limited capacity. during pre-training, they cannot fully capture the vast and continuously evolving body of real-world knowledge @xcite . as a result, slms often struggle in real-world scenarios where accurate responses depend on newly emerging or dynamically updated information. retrieval-augmented generation (rag) @xcite mitigates this limitation by retrieving a top-k set of semantically relevant documents at inference time, which are then conditioned upon during response generation. by decoupling knowledge retrieval from parametric memory, rag enhances the adaptability of slms, improving response fidelity while obviating the need for expensive and frequent model retraining @xcite @xcite . however, one inherent challenge for rag pipelines is the imperfect retrieval information. the top-k documents returned by a retriever may include irrelevant or noisy information, which can mislead small-scale llms that are not robust enough to effectively filter out such distractions. as shown in fig. 1 , the susceptibility of slms to retrieval noise highlights a critical bottleneck in rag pipelines. this limitation highlights the necessity of developing more robust mechanisms to enhance slms' resilience against spurious retrieval artifacts and improve their reliability in downstream tasks. several existing works have sought to enhance the robustness of rag systems. prompting-based approaches @xcite attempt to mitigate noise by instructing the model to disregard irrelevant information during inference. however, these techniques heavily depend on the model's intrinsic ability to distinguish useful from spurious content-a capability that slms often lack due to their limited reasoning and generalization capacity. on the other hand, fine-tuning-based strategies @xcite aim to improve retrieval robustness by training models on curated, denoised datasets using supervised finetuning (sft). unfortunately, sft tends to mimic the behavior present in the training data, making it highly sensitive to noise and prone to overfitting @xcite , which ultimately limits the generalization ability of the model. in light of these challenges, we propose robust retrieval-augmented generation with small llms via margin-aware preference optimization (roserag). roserag introduces a novel framework that robustly aligns slms with high-quality responses without distillation from teacher llms through a three-stage process: preference data generation, contrastive preference data selection, and margin-aware preference optimization. specifically, we employ a multi-turn prompting strategy to generate detailed rationales, coupled with rejection sampling @xcite to filter out spurious reasoning, thereby mitigating the influence of noisy retrieved documents. besides, our contrastive selection mechanism identifies the most challenging response pairs, enabling the model to explicitly maximize the margin between preferred and non-preferred outputs. by integrating these components into a unified optimization framework, roserag significantly improves the reliability and accuracy of small-scale llms in retrieval-augmented settings, especially under imperfect retrieval conditions that mirror real-world scenarios. our contributions are summarized as follows: ## related work retrieval-augmented generation (rag) enhances large language models (llms) by integrating non-parametric knowledge to improve generation quality. early approaches @xcite treat llms as passive consumers of retrieved information, assuming that retrieval inherently improves generation. however, recent studies @xcite @xcite highlight that retrieved knowledge is often noisy, incomplete, or even misleading, which can hurt rather than enhance the performance llms. to better align the retrieved information with llms, @xcite ; @xcite propose initial assessments to determine whether retrieval is necessary, using either bert-family models or the frozen llm as a classifier. however, these approaches depend on classifier accuracy and does not improve the llm's inherent ability to handle noisy information. blendfilter @xcite , astute @xcite , and robus-trag @xcite leverage the llm itself to analyze or filter irrelevant information. however, their effectiveness is contingent on the model's capability; small-scale llms often fail to achieve reliable results due to their limited intelligence. another line of research @xcite @xcite investigates training llms to handle noisy contexts, e.g., those retrieved from external corpora. these approaches typically leverage powerful models such as gpt-4 or require extensive labeled data from auxiliary tasks to generate high-quality responses. however, such reliance on large-scale models and costly annotations limits scalability and practicality, particularly for resource-constrained applications. very recently, instructrag @xcite instructs llms to provide rationales linking answers to retrieved passages, but pure supervised fine-tuning cannot fully unleash the model's reasoning capability towards rag applications. knowpo @xcite and dpa-rag @xcite optimize model preferences to improve noisy information analysis. yet, they still require access to powerful llms (e.g. gpt-4) to create preference data. unlike existing approaches, roserag is specifically designed to enhance the robustness of slms against noisy retrieved information through margin-aware preference optimization, eliminating the need for additional classifiers or high-resource llms. ## preliminary in this section, we introduce the concept of standard retrieval-augmented generation (rag) and monolithic preference optimization (orpo). ## method our roserag enhances retrieval-augmented generation for slms through margin-aware preference optimization. it consists of three key stages, as shown in fig. 2: (1) preference data generation, where the model is prompted with retrieved knowledge and ground-truth answers to generate rationales, filtered via rejection sampling; ## theoretical analysis in this section, we provide a theoretical analysis to support the effectiveness of roserag. we derive a closed-form solution for the model learned by roserag and examine how the proposed preference selection strategy influences the model's behavior. we begin with a regularity condition ensuring that the optimization space is sufficiently expressive to achieve the global optimum. assumption 5.1. assume that p (y|x) belongs to the optimization space {p θ : θ ∈ θ} such that lemma 5.1. under assumption 5.1, the solution to optimizing eqn. (4) is to illustrate the benefit of the proposed preference selection strategy, we consider a scenario where y w and y l are random variables following we compare our method to a baseline that applies orpo without preference selection (i.e., @xmath0. (3)). let pθ (y|x) denote the solution obtained without selection. we measure the response quality using the absolute distance from the expected preferred response, l(y) = |y -e[y w ]|. the following theorem formalizes the advantage of our method. the proof is deferred to app c. theorem 5.1. under (6), assume pθ (y|x) generates y with density function: let y ∼ p θ (y|x) and ỹ ∼ pθ (y|x). then, the expected absolute distance under p θ is smaller than that under pθ : theorem 5.1 suggests that the proposed preference selection strategy improves response alignment with the preferred choice y w . our analysis reveals that this selection process amplifies the gap between preferred and non-preferred responses, compelling the model to enhance their separation and ultimately leading to more accurate and reliable outputs. ## experiment in this section, we extensively evaluate the proposed roserag and answer the following questions: rq1) how does roserag perform compared to state-of-the-art baselines? rq2) what are the roles of rejection sampling and preference data selection in model performance improvements respectively? rq3) can the proposed roserag benefit from more retrieved documents? rq4) is it possible to apply different preference optimization method to roserag? rq5) how does the performance change with varying β? 6.1 datasets and experiment settings datasets and evaluation metrics. we conduct experiments on three public benchmarks, including hotpotqa @xcite , 2wikimulti-hopqa @xcite , and strategyqa @xcite . following shao et al. (2023); wang et al. (2024b) , we evaluate the first 500 questions from the development dataset for hotpotqa and 2wikimultihopqa, and evaluate questions from the development dataset for strategyqa. for multihop question answering datasets, we employ exact match (em) and f1 as evaluation metrics, and for the commonsense reasoning dataset, which is a binary classification task, we use accuracy and f1 score as the metrics. to evaluate the retrieval performance, we leverage widely used recall as the evaluation metric. baselines. we adopt following state-of-the-art baselines to evaluate against roserag: 1) cot prompting @xcite , 2) react @xcite , 3) selfask @xcite , 4) blendfilter @xcite , 5) instructrag @xcite , 6) retrobust (yoran et al., 2024), 7) as-tute @xcite , and 8) icl+rag @xcite . we show more detailed information about baselines in the appendix a. implementation details. we evaluate models with three small-sclae llms: qwen2.5-1.5b-instruct @xcite , llama-3.2-1b-instruct 1 , and gemma-2-2b-it @xcite . we utilize the state-of-the-art efficient retrieval method colbert v2 @xcite as the retriever implemented by @xcite . the knowledge base we employ is the collection of wikipedia abstracts dumped in 2017 @xcite . we show the detailed information about our implementations in the appendix b. ## conclusion we introduce roserag, a novel framework that enhances the robustness of small-scale llms (slms) in retrieval-augmented generation through margin-aware preference optimization. by leveraging multi-turn prompting with rejection sampling and contrastive preference data selection, our approach effectively mitigates the impact of noisy retrieved content. we conducted extensive experiments on three benchmarks, and the results demon-strate that roserag outperforms state-of-the-art baselines. moreover, roserag can be generalized well for different kinds of slms, justifying its generalizability and broad applicability for rag.