R ^2 AG : Incorporating Retrieval Information into Retrieval Augmented Generation (2024)

Abstract: Retrieval augmented generation (RAG) has been applied in many scenarios to augment large language models (LLMs) with external documents provided by retrievers. However, a semantic gap exists between LLMs and retrievers due to differences in their training objectives and architectures. This misalignment forces LLMs to passively accept the documents provided by the retrievers, leading to incomprehension in the generation process, where the LLMs are burdened with the task of distinguishing these documents using their inherent knowledge. This paper proposes R ^2 AG, a novel enhanced RAG framework to fill this gap by incorporating **R**etrieval information into **R**etrieval **A**ugmented **G**eneration. Specifically, R ^2 AG utilizes the nuanced features from the retrievers and employs a R ^2 -Former to capture retrieval information. Then, a retrieval-aware prompting strategy is designed to integrate retrieval information into LLMs’ generation. Notably, R ^2 AG suits low-source scenarios where LLMs and retrievers are frozen. Extensive experiments across five datasets validate the effectiveness, robustness, and efficiency of R ^2 AG. Our analysis reveals that retrieval information serves as an anchor to aid LLMs in the generation process, thereby filling the semantic gap.

## introduction retrieval augmented generation (rag) @xcite significantly enhances the capabilities of large language models (llms) by integrating external, non-parametric knowledge provided by retrievers. in rag framework, the retriever locates and looks up useful documents based on a given query, and then the llm interacts with these retrieved results to generate a response. the coordination of retrieval and generation achieves impressive performance without additional training. especially in domain-specific and knowledge-intensive r 2 ag employs a trainable r 2 -former to bridge the semantic gap between retrievers and llms. optionally, llms can be fine-tuned to understand the retrieval information further. however, there exists a semantic gap between retrievers and llms due to their vastly different training objectives and architectures @xcite . specifically, retrievers, typically encoder architecture, are designed to retrieve the most relevant documents for a query @xcite . conversely, llms, generally decoder architecture, are expected to answer questions based on their inherent knowledge or given documents. however, the interaction between retrievers and llms in rag primarily relies on simple text concatenation @xcite . this poor communication strategy will lead to several challenges for llms. externally, it is hard for llms to utilize more information from retrievers in separate processes. in rag, the retrieved documents that only preserve sequential relationships are unidirectionally delivered to llms, and llms do not fully understand why retrievers provide the documents. particularly, low-quality documents inevitably appear in retrieved results @xcite , but llms have to accept this noise passively. internally, it is hard for llms to handle all of the retrieved documents with their inherent knowledge. llms must process all the results and assess which documents are important, impacting their ability to generate accurate answers @xcite . moreover, llms face the lost-in-middle problem in overly long documents @xcite , leading to further misunderstanding. unfortunately, existing enhanced rag methods, including pre-processing approaches @xcite @xcite and compression-based approaches @xcite @xcite , do not recognize this semantic gap between retrievers and llms. they remain to treat retrieval and generation as separate processes and directly add processed or compressed documents into the inputs for llms. these strategies ignore the semantic connections necessary for deeper comprehension, which may lead to potentially misleading llms even with perfect retrievers. to address these challenges, it is essential to bridge the semantic gap between retrievers and llms. as previously mentioned, retrievers can provide high-quality semantic representations that can be beneficial for catching nuanced differences among documents @xcite . thus, our intuition is to exploit these semantic representations as additional knowledge, empower llms to gain a deeper comprehension of the retrieved documents, and thereby generate more accurate responses. this paper proposes a cost-effective enhanced rag framework to incorporate retrieval information into retrieval argumented generation (named r 2 ag), enhancing llms' perception of the key information among retrieved documents. specifically, r 2 ag adopts an input processing pipeline that transforms semantic representations from a retriever into unified retrieval features. then, a trainable r 2 -former is employed to capture essential retrieval information. as shown in figure 1, r 2 -former is a pluggable and lightweight model placed between the retriever and the llm. finally, through a retrieval-aware prompting strategy, the llm receives additional embeddings that contain retrieval information. this strategy aligns the knowledge from retrievers with llms without changing the content and order of retrieved documents, thereby relieving information loss. r 2 ag offers the flexibility to fine-tune r 2 -former alone or both with llms. thus, in r 2 ag framework, both retrievers and llms can be frozen to save computational costs, making r 2 ag suitable for scenarios with limited resources. overall, our contributions are summarized as follows: experimental results demonstrate the superior performance and robustness of r 2 ag in various scenarios. our analysis shows that r 2 ag increases latency by only 0.8% during inference. furthermore, it demonstrates that retrieval information anchors llms to understand retrieved documents and enhances their generation capabilities. ## r 2 ag 3.1 problem formulation and overview rag involves the task that aims to prompt an llm to generate answers based on a query and documents returned by a retriever. formally, given a query q and a list of documents @xmath0, d 2 , • • • , d k } in preference order ranked by the retriever f r , the llm, a generator f g , is expected to generate the output ŷ. the pipeline can be expressed as: figure 2 illustrates the overall framework of r 2 ag. initially, given a query and retrieved documents, r 2 ag processes representations modeled by a retriever into unified-format features. these list-wise features consider nuanced relationships both between the query and documents and among the documents themselves. then, a r 2 -former is designed to capture retrieval information for llm usage. it allows unified features to interact with each other via self-attention mechanism, enabling it to understand complex dependencies. to integrate retrieval information into the llm's generation process, r 2 ag adopts a retrieval-aware prompting strategy to insert the retrieval information into the llm's input embedding space without causing information loss or increasing much complexity. besides, r 2 ag is flexible to be applied in low-source scenarios where llms are frozen. ## conclusion and future work this paper proposed a novel enhanced rag framework named r 2 ag to bridge the semantic gap between the retrievers and llms. by incorporating retrieval information from retrievers into llms' generation process, r 2 ag captures a comprehensive understanding of retrieved documents. experimental results show that r 2 ag outperforms other competitors. in addition, the robustness and effectiveness of r 2 ag are further confirmed by detailed analysis. in future work, more retrieval features could be applied to r 2 ag framework.