CORD : Balancing CO nsistency and Rank Distillation for Robust Retrieval-Augmented Generation (2025)

Abstract: With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation: CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks.

## introduction recently, large language models (llms) have incorporated retrievers to augment input contexts for more grounded generation. however, during retrieval-augmented generation (rag), llms reportedly suffer from position bias where they pay disproportionate attention to different parts, worsened as the input becomes longer @xcite ). an existing solution has synthesized a training set by randomizing the position of gold segment @xcite . it allows llms to implicitly learn that relevant information can appear at any position, mitigating position bias. our distinction is to pursue dual goals of (1) consistency for mitigating position bias and (2) rank distillation, learning to utilize meaningful signals in the given order from the retriever and also to denoise it, for robust rag. for co, we extend the position-perturbing training intuition, by augmenting the retriever-given order of contexts c with its perturbation c foot_0 , sharing the same ground truth ŷ. green arrows in figure 1 visualize how this augmentation indirectly enforces consistency by guiding predictions y from c and y 1 from c 1 , to converge to the ground-truth ŷ. to further enforce consistency, a distillation loss can be added to directly penalize the distributional divergence in all outputs. the blue arrow in figure 1 visualizes this loss further incentivizing consistent internal representation, by distilling 'dark knowledge' @xcite @xcite ) from one to another. however, pursuing co objective alone, without balancing it with the rd objective, is counterproductive in some scenarios as illustrated in our technical contribution is to adapt c 1 to the given scenario, by controlling the degree of perturbation, in place of c 1 with a fixed randomization. we define an interpolated space of perturbations and dynamically sample an appropriate level of perturbation from it. our contribution can be summarized as follows: @xcite we propose cord, balancing connsistency and rank distillation in rag. (2) we show distilling with a controlled perturbation, sampled from an interpolated space of teachers, is effective across 5 diverse rag scenarios, whereas existing consistency methods fall short. our distinction is to use position perturbation for a different objective of data augmentation for consistency training. ## method 3.1 co: consistency regularization we propose to mitigate position bias by regularizing output consistency over possible perturbations, through (1) augmentation and (2) distillation. first, we explain how augmenting positionperturbed examples contributes to consistency. we first formalize rag as generating an answer y given an input x, inspired by @xcite , referred to as in2, we employ position perturbation to augment c from the corpus c with c 1 . for comparison, in2 synthesized question and context pq, cq pairs where the gold passage s for generating the gold answer ŷ appears in various positions. as figure 2 shows, we retain both the original pq, c, ŷq and the perturbed examples pq, c 1 , ŷq: unlike in2's using c 1 only for training (orange arrows), we train over the augmented dataset c 1 which includes both c and c 1 (blue arrows). predictions for both are supervised to converge to the same ground-truth ŷ using the loss in eq. 2. second, by adding a distillation loss, we can further match token-level output probability distributions for c and c 1 . we use the sum of jensen-shannon divergence (jsd) between output proba-bility distributions at each time step t for this purpose: foot_1 finally, the two types of loss in eq. 2 and 3 can be combined to obtain our training objective: ## results we design evaluations to answer these research questions: ## conclusion we have presented cord, to balance the tension between co (consistency) and rd (rank distillation) objectives in rag. for the former, we augment order-perturbed contexts and add distillation loss for explicit consistency regularization. for the latter, cord adaptively chooses desirable degree of perturbation to prevent unlearning valuable prior from the retriever. cord consistently outperforms existing methods in diverse rag scenarios.