DRAGIN : Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models (2024)

Abstract: Dynamic retrieval augmented generation (RAG) paradigm actively decides when and what to retrieve during the text generation process of Large Language Models (LLMs).There are two key elements of this paradigm: identifying the optimal moment to activate the retrieval module (deciding when to retrieve) and crafting the appropriate query once retrieval is triggered (determining what to retrieve).However, current dynamic RAG methods fall short in both aspects. Firstly, the strategies for deciding when to retrieve often rely on static rules. Moreover, the strategies for deciding what to retrieve typically limit themselves to the LLM’s most recent sentence or the last few tokens, while the LLM’s information needs may span across the entire context.To overcome these limitations, we introduce a new framework, DRAGIN, i.e., Dynamic Retrieval Augmented Generation based on the Information Needs of LLMs. Our framework is specifically designed to make decisions on when and what to retrieve based on the LLM’s information needs during the text generation process.We evaluate DRAGIN along with existing methods comprehensively over 4 knowledge-intensive generation datasets. Experimental results show that DRAGIN achieves superior performance on all tasks, demonstrating the effectiveness of our method.

## introduction in recent years, large language models (llms) have made significant advancements across various natural language processing (nlp) tasks, quickly becoming a critical element in numerous ai applications @xcite @xcite @xcite . despite their impressive capabilities, these models often produce text that seems coherent and plausible but factually incorrect, a problem commonly known as hallucination @xcite @xcite @xcite . to mitigate this issue, retrieval-augmented generation (rag) has emerged as a prominent solution. rag enhances llms by retrieving and incorporating relevant information from external databases into the llms' inputs. it has demonstrated superior effectiveness across numerous nlp challenges @xcite @xcite @xcite @xcite . traditional methods of rag typically rely on single-round retrieval, using the llm's initial input to retrieve relevant information from external corpora. while this method is effective for straightforward tasks, it tends to fall short for complex multi-step tasks and long-form generation tasks @xcite . in contrast, dynamic rag @xcite @xcite performs multiple times of retrieval during the generation process of llms. it includes two steps: identifying the optimal moment to activate the retrieval module (deciding when to retrieve), and crafting the appropriate query once retrieval is triggered (determining what to retrieve). depending on when and what to retrieve, a variety types of methods have been proposed in this direction. for example, ir-cot @xcite adopts a global augmentation method where retrieval is conducted for each generated sentence, with the latest generated sentence used as the query. retro @xcite and ic-ralm @xcite define a sliding window and trigger the retrieval module based on a preset number of processed tokens, and the last n tokens are used as the query. however, existing dynamic rag methods face several critical challenges, primarily in determining the optimal timing for retrieval and formulating effective queries when retrieval is triggered. first of all, existing approaches often rely on static rules to decide when to retrieve, neglecting the assessment of necessity and potential risks involved. on the one hand, depending on the quality of the input query and retrieval models, unnecessary retrieval augmentation may introduce irrelevant or noisy data to llms which could jeopardize the quality of the outputs. on the other hand, conducting retrieval augmentation will inevitably increase the time and computation cost of llm inference, such cost is unworthy if llms can generate correct outputs by themselves. additionally, the strategies of existing studies in determining what to retrieve often restrict themselves to the llm's most recent sentence or the last few tokens. this approach may not capture the model's real-time information needs since the llm's information needs may actually be related to terms that span the entire context. retrieving documents in this manner is thus suboptimal in many cases. to overcome these limitations, we introduce a new framework, dragin, i.e., dynamic retrieval augmented generation based on the information needs of llms. our framework is specifically designed to make decisions on when and what to retrieve, based on the llm's information needs during the text generation process. for the timing of retrieval, we propose rind: real-time information needs detection, which considers the llm's uncertainty about its own generated content, the influence of each token on subsequent tokens, and the semantic significance of each token. for the formulation of retrieval queries, we propose qfs: query formulation based on self-attention, which innovates query formulation by leveraging the llm's self-attention across the entire context. dragin is a lightweight rag framework that can be incorporated into any transformer-based llms without further training, fine-tuning, or prompt engineering. we comprehensively evaluate dragin along with existing dynamic rag frameworks over four knowledge-intensive generation benchmarks. experimental results show that dragin achieves superior performance on all datasets, demonstrating the effectiveness of our method. moreover, the results of the ablation study indicate that our proposed new strategies for "when to retrieval" (i.e., rind) and "what to retrieval" (i.e., qfs) perform uniformly better than other strategies in existing rag methods despite retrieval models and llms. in summary, the contributions of our paper are as follows: ## methodology in this section, we introduce the dragin framework in detail. dragin consists of two components: real-time information needs detection (rind) and query formulation based on selfattention (qfs), as illustrated in figure 1 . we introduce rind in section 3.1, and qfs in sec-tion 3.2. ## conclusions and future works in this work, we propose dragin, a dynamic rag framework tailored to address the real-time information needs of llms during text generation. by integrating rind for timely retrieval activation and qfs for precise query formulation, dragin significantly outperforms existing dynamic rag methods across various knowledge-intensive benchmarks.