Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps (2025)

Abstract: Retrieval-augmented generation (RAG) has emerged as a pivotal method for expanding the knowledge of large language models. To handle complex queries more effectively, researchers developed Adaptive-RAG (A-RAG) to enhance the generated quality through multiple interactions with external knowledge bases. Despite its effectiveness, A-RAG exacerbates the pre-existing efficiency challenges inherent in RAG, which are attributable to its reliance on multiple iterations of generation. Existing A-RAG approaches process all retrieved contents from scratch. However, they ignore the situation where there is a significant overlap in the content of the retrieval results across rounds. The overlapping content is redundantly represented, which leads to a large proportion of repeated computations, thus affecting the overall efficiency. To address this issue, this paper introduces a model-agnostic approach that can be generally applied to A-RAG methods, which is dedicated to reducing the redundant representation process caused by the overlapping of retrieval results. Specifically, we use cache access and parallel generation to speed up the prefilling and decoding stages respectively. Additionally, we also propose an instruction-driven module to further guide the model to more effectively attend to each part of the content in a more suitable way for LLMs. Experiments show that our approach achieves 2.79 and 2.33 times significant acceleration on average for prefilling and decoding respectively while maintaining equal generation quality.

## introduction work smarter, not harder. ## methods figure 2 details the workflow of idr 2 , and we divide the internal process of each iteration round in a-rag into three phases: retrieval, prefilling, and decoding. when processing a query, cached document representations in the cics module are first verified. these representations are loaded and combined with uncached documents for prefilling, where the idgr module guides llms to prioritize relevant content while filtering noise. before each autoregressive step, matching subsequent phrase fragments are queried in the igpg module to enable parallel generation. ultimately, these approaches enhance the overall efficiency of a-rag. details of each module are given respectively in the remainder of this section. ## conclusion this paper presents idr 2 , a model-agnostic approach that significantly improves the efficiency of a-rag by addressing redundant representation processing across multiple iterations. to prevent redundant computation, this paper introduces the cics module for caching document representations. the idgr module then guides the generation process to focus on crucial information through the instruction-following capability of llms. furthermore, the igpg module leverages the corre-lation between generated content and documents to enable parallel generation of existing content, reducing autoregressive rounds. through extensive experiments, idr 2 consistently demonstrates remarkable efficiency improvements, achieving up to 2.0× acceleration while maintaining generation quality. our future work aims to further compress the kv cache while preserving its representation capabilities.