Unified Active Retrieval for Retrieval Augmented Generation (2024)

Abstract: In Retrieval-Augmented Generation (RAG), retrieval is not always helpful and applying it to every instruction is sub-optimal. Therefore, determining whether to retrieve is crucial for RAG, which is usually referred to as Active Retrieval. However, existing active retrieval methods face two challenges: 1. They usually rely on a single criterion, which struggles with handling various types of instructions. 2. They depend on specialized and highly differentiated procedures, and thus combining them makes the RAG system more complicated and leads to higher response latency. To address these challenges, we propose Unified Active Retrieval (UAR). UAR contains four orthogonal criteria and casts them into plug-and-play classification tasks, which achieves multifaceted retrieval timing judgements with negligible extra inference cost. We further introduce the Unified Active Retrieval Criteria (UAR-Criteria), designed to process diverse active retrieval scenarios through a standardized procedure. Experiments on four representative types of user instructions show that UAR significantly outperforms existing work on the retrieval timing judgement and the performance of downstream tasks, which shows the effectiveness of UAR and its helpfulness to downstream tasks.

## introduction with the rapid development of large language models (llms) @xcite @xcite @xcite , ai assistants based on llms become unbiquitous and show remarkable abilities on various types of instructions, e.g., coding, writing and reasoning @xcite @xcite @xcite @xcite . however, llms often generate fabricated and nonfactual information @xcite ## methodology uar is a plug-and-play active retrieval framework. as shown in figure 2 , we fix the parameters of the llm and train a lightweight classifier for each active retrieval criteria using the model's hidden states, which is far more efficient than fine-tuning the entire model. besides, uar determines the need for active retrieval following the uar-criteria shown on the right side of figure 2 , invoking retrieval when necessary and avoiding unnecessary across various scenarios, making rag more effective and efficient. for instructions requiring retrieval, we append the retrieved documents to the ## analysis different scenarios have varying levels of discrimination difficulty. as shown in to improve the performance bottleneck of the self-aware classifier, we attempt to fine-tune the entire large language model as the classifier. from the results in ## conclusion in this paper, we introduce uar, a unified active retrieval framework for retrieval-augmented generation. unlike existing methods that rely on a single criterion, uar incorporates four orthogonal criteria into plug-and-play classification tasks, enabling comprehensive retrieval timing judgments with minimal inference cost and no loss of model capabilities. we also introduce uar-criteria for processing various active retrieval scenarios uni-formly. we curate the active retrieval benchmark (ar-bench) to assess the retrieval timing accuracy of active retrieval methods across different scenarios. experimental results demonstrate that uar significantly outperforms existing methods on ar-bench and downstream tasks, highlighting its effectiveness and benefits to downstream applications.