I mp RAG : Retrieval-Augmented Generation with Implicit Queries (2025)

Abstract: Retrieval-Augmented Generation (RAG) systems traditionally treat retrieval and generation as separate processes, requiring explicit textual queries to connect them. This separation can limit the ability of models to generalize across diverse tasks. In this work, we propose a query-free RAG system, named ImpRAG, which integrates retrieval and generation into a unified model. ImpRAG allows models to implicitly express their information needs, eliminating the need for human-specified queries. By dividing pretrained decoder-only language models into specialized layer groups, ImpRAG optimizes retrieval and generation tasks simultaneously. Our approach employs a two-stage inference process, using the same model parameters and forward pass for both retrieval and generation, thereby minimizing the disparity between retrievers and language models. Experiments on 8 knowledge-intensive tasks demonstrate that ImpRAG achieves 3.6-11.5 improvements in exact match scores on unseen tasks with diverse formats, highlighting its effectiveness in enabling models to articulate their own information needs and generalize across tasks. Our analysis underscores the importance of balancing retrieval and generation parameters and leveraging generation perplexities as retrieval training objectives for enhanced performance.

## introduction retrieval-augmented generation (rag; @xcite @xcite typically involves two key operations: retrieval and generation. rag systems retrieve relevant information to enhance generation models, enabling them to respond more effectively to prompts by providing long-tail knowledge or up-to-date information. while effective, traditional approaches figure 1 : diagram illustrating the inference process of imprag on the entity linking task. we divide decoderonly llms into three layer groups for specialized finetuning: bottom (green), middle (red), and top (blue). the bottom layers are optimized for retrieval tasks. the middle and top layers handle the reading of retrieved passages, with cross-attention disabled in the top layers to reduce memory consumption. standard rag systems would require a task-specific design of queries (e.g., use the substring "british" as the query in the shown example). in contrast, imprag uses implicit queries, eliminating the need for explicit specification of queries and allowing models to generalize across unseen tasks with varied formats. to achieve this, we introduce imprag, a novel approach that integrates retrieval and generation into a unified model and process. this allows models to convey their own information needs implicitly, reducing the need for prior knowledge of test tasks and for humans to formulate explicit textual queries in advance. at its core, imprag aims to enable retrieval capabilities through retrieval heads in self-attention. building upon pretrained decoderonly language models, imprag divides the layers into three groups: the bottom group for retrieval and the middle and top groups for reading and generation. figure 1 illustrates an example of applying im-prag to the entity linking task, where models are tasked with linking the mention "british" to an entity in wikipedia, given the context paragraph. a typical rag model would require the design of a separate query template, such as using only the mention text, to achieve reasonable retrieval performance. in contrast, imprag uses implicit queries and can perform retrieval and generation jointly without the need for additional template design, making it more generalizable. during training, we optimize two objectives simultaneously: generation loss and retrieval loss. the generation loss is the standard causal language modeling loss, while the retrieval loss first utilizes pseudo labels generated by trained retrievers to warm up the retrieval ability and then self-improves using its own generation log likelihood for the remainder of the training. at inference time, we employ a two-stage process. first, we embed passages using the bottom layer for retrieval, and then utilize the top layer group to read the retrieved passages and generate the final responses. by leveraging the same forward pass and model parameters for both retrieval and generation, imprag reduces the disparity between retrievers and llms. in experiments, we train models on datasets that either require retrieval or do not. the datasets requiring retrieval are used to enhance retrieval performance, while those not requiring retrieval are used to improve models' instruction-following capabilities. we evaluate the models on 8 knowledge-intensive tasks, focusing on different aspects: basic question answering, multihop reasoning, and instruction following. we also establish strong baselines that perform rag in the retrieve-thengenerate paradigm, including ra-dit @xcite , a method that iteratively updates llms and retrievers to better align the two. our experiments demonstrate that imprag achieves slightly better performance on 4 tasks with formats similar to the training tasks, with an improvement of 0.2-0.6 points in exact match scores, all without the need for additional model parameters. moreover, it significantly outperforms previous approaches on unseen test tasks with more diverse formats, achieving improvements of 3.6-11.5 points in exact match scores and 5.0-23.2 points in retrieval recalls. this highlights the effectiveness of enabling models to articulate their own information needs. our analysis indicates that carefully selecting layer group boundaries that balance the parameters used for retrieval and generation, using both trained retrievers for warmup and then selfimprove by leveraging generation perplexities as retrieval training objectives, and instruction tuning training datasets is crucial for achieving superior performance in imprag. our analysis also reveals that imprag is effective in transferring supervision from generation tasks to retrieval tasks, showing the potential of using an unified model architecture for performing retrieval and generation jointly. ## related work there has been a lot of work on using the retrievethen-generate paradigm for rag @xcite . many efforts in this line of work have focused on optimizing retrievers using training signals from generation models, and optionally, the reverse @xcite @xcite . although the specifics can differ, these approaches generally utilize distinct models and input templates for the retrieval and generation phases. a closely related study is that of @xcite , which seek to use the same model for retrieval and generation. however, their research primarily focuses on encoder-decoder style models and their models still rely on separate input templates for retrieval and generation. another related work by @xcite explores the use of special tokens for retrieval, but their study emphasizes in-domain task performance rather than unseen task generalization. this work is also related to research on query formulation in the context of multihop question answering, where previous studies typically generate textual queries by prompting llms, followed by retrieval using a separate retriever @xcite @xcite @xcite . @xcite enable llms to generate textual queries through synthetic data generation. additionally, this work is connected to memory architectures in rag @xcite , which aim to utilize the key-value (kv) caches of llms to reduce computational costs, rather than focusing on minimizing the disparities between generation and retrieval. another relevant area of research is instruction tuning for rag. lin et al. (2024) perform instruction tuning for both retrievers and llms and then align them through iterative updates. @xcite conduct instruction tuning for retro-like models @xcite . @xcite align retrievers with llms using synthetic data generated by llms. unlike our work, these studies still treat retrieval and generation as separate processes. in a similar vein, researchers have tried to teach retrievers to follow instructions for building general-purpose information retrieval systems @xcite @xcite . since imprag enables its retrieval capabilities by using self-attention, it is related to research on investigating retrieval heads in the context of long context llms @xcite . ## method we build on an autoregressive pretrained language model and enable it to perform retrieval and generation jointly. our model, imprag, is based on the llama 3 family @xcite , with architectural modifications to support retrieval and retrieval-augmented generation. at a high level, the layer grouping strategy of imprag is inspired by the observation that llms learn distinct functions at different layers @xcite . consequently, we have designed the layer groups to align with the capabilities required for retrievalaugmented generation, i.e., retrieval and generation. ## conclusion we present imprag, a query-free retrievalaugmented generation (rag) system that implicitly captures information needs without requiring human-specified queries. unlike prior work that treats retrieval and generation as separate components with independently trained models, im-prag unifies them within a single decoder-only language model by partitioning it into specialized layer groups and jointly optimizing for both retrieval and generation. the same model parameters and forward pass are shared across retrieval and generation, effectively minimizing the mismatch between the retriever and the generator. im-prag demonstrates strong performance across eight knowledge-intensive tasks, outperforming traditional rag systems and delivering substantial gains on unseen tasks with diverse formats.