R etrieval QA : Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering (2024)

Abstract: Adaptive retrieval-augmented generation (ARAG) aims to dynamically determine the necessity of retrieval for queries instead of retrieving indiscriminately to enhance the efficiency and relevance of the sourced information. However, previous works largely overlook the evaluation of ARAG approaches, leading to their effectiveness being understudied. This work presents a benchmark, RetrievalQA, comprising 1,271 short-form questions covering new world and long-tail knowledge. The knowledge necessary to answer the questions is absent from LLMs; therefore, external information must be retrieved to answer correctly. This makes RetrievalQA a suitable testbed to evaluate existing ARAG methods. We observe that calibration-based methods heavily rely on threshold tuning, while vanilla prompting is inadequate for guiding LLMs to make reliable retrieval decisions. Based on our findings, we propose Time-Aware Adaptive Retrieval (TA-ARE), a simple yet effective method that helps LLMs assess the necessity of retrieval without calibration or additional training.

## introduction retrieval-augmented generation (rag) @xcite @xcite that augments large language models (llms) with retrieval of relevant information has become increasingly popular in knowledge-intensive tasks, including open-domain question-answering (qa) @xcite @xcite . however, standard rag methods conduct retrieval indiscriminately, irrespective of the input query, which may result in suboptimal task performance and increased inference costs @xcite . on one hand, llms figure 1 : above: qa accuracy on our retrievalqa w/, w/o retrieval, and adaptive retrieval. we set threshold @xmath0.5 for calibration-based self-rag @xcite and use model-based vanilla prompting for others ( §3). we find that self-rag requires threshold tuning to balance qa performance and retrieval efficiency, while vanilla prompting is insufficient in guiding llms to make reliable retrieval decisions ( §3.3). below: an error analysis for gpt-3.5. at least half of the time, gpt-3.5 is unaware that it needs retrieval (i.e., red area, §3.4). to alleviate the limitations of rag mentioned above, recent studies advocate for adaptive rag (arag), which dynamically determines retrieval necessity and relies only on llms' parametric knowledge when deemed unnecessary @xcite . however, the effectiveness of these meth-ods is understudied, as there is no suitable benchmark and evaluation. arag approaches can be categorized into calibration-based and model-based judgement. calibration-based methods @xcite @xcite , while effective, trigger retrieval only when a metric surpasses a pre-defined threshold. for example, @xcite heuristically retrieve when the popularity of an entity on wikipedia is below a certain threshold; @xcite trigger retrieval if any token in the temporarily generated sentence has low confidence. clearly, these ad-hoc calibrationbased methods are suboptimal, as we need to tune thresholds for different datasets and models to balance task performance and inference overheads. to obviate the hyperparameter threshold, model-based methods @xcite directly prompt llms for retrieval decisions, given the observation that llms can acknowledge their knowledge boundaries to some extent @xcite . these methods undergo separate evaluations, and their effectiveness remains ambiguous due to the limited scope of the assessments. in this paper, we investigate to what extent llms can perform calibration-free adaptive retrieval via prompting. to answer this question, we need to evaluate whether llms retrieve only when necessary. this requests a benchmark that distinguishes between questions that can be answered using llms' parametric knowledge and those that require external information through retrieval. nevertheless, commonly used open-domain qa datasets @xcite @xcite fail to fulfil this purpose, as various llms have distinct sizes and levels of pre-trained knowledge, making them inadequately assess the necessity of external retrieval for llms. to fill this gap, we create retrievalqa, a shortform qa dataset, covering new world and long-tail knowledge and spanning diverse topics. we ensure the knowledge necessary to answer the questions is absent from llms. therefore, llms must truthfully decide whether to retrieve to be able to answer the questions correctly. retrievalqa enables us to evaluate the effectiveness of arag approaches, an aspect predominantly overlooked in prior studies and recent rag evaluation systems @xcite @xcite , which focus only on task performance, the relevance of retrieval context or the faithfulness of an-swers. using retrievalqa as a testbed, we benchmark both calibration-based and model-based methods with varying sizes of llms. as shown in fig. 1 , we find calibration-based self-rag requires threshold tuning to balance qa performance and retrieval efficiency, while vanilla prompting is insufficient in guiding llms to make reliable retrieval decisions. as an initial effort, we propose time-aware adaptive retrieval (ta-are), a simple yet effective method to improve arag via in-context learning (icl; @xcite ), obviating the need for calibration or additional training. to sum up, this paper makes the following contributions: 1 we create a new dataset retrievalqa to assess arag for short-form open-domain qa; 2 we benchmark existing methods and conduct extensive analysis, finding that vanilla prompting is insufficient in guiding llms to make reliable retrieval decisions; 3 we then propose ta-are, a simple yet effective method to help llms assess the necessity of retrieval without calibration or additional training. filtering out answerable questions. as discussed in §1, to ensure the questions cannot be answered without external knowledge, we conduct strict filtering. to save manual work, we prompt gpt-4 for answers in a closed-book qa setting without access to external knowledge (see prompt template fig. 7 in appendix). then, we calculate the token-level f1 scores @xcite and remove questions that have shared tokens between the prediction and the ground truth, i.e., only keep questions with @xmath1. our rationale is that if state-of-the-art gpt-4 cannot answer correctly without retrieval, weaker llms are also highly likely to fail. finally, after filtering, we have obtained 1,271 out of 9,336 questions, covering new world and long-tail knowledge and spanning diverse topics. to avoid potential bias in the evaluation towards methods that retrieve more often, we additionally collect 1,514 questions that can be answered using gpt-2's parametric knowledge from the discard set. more details are in the appendix a.2. we conduct a sanity check using various sizes of llms in fig. 1 and in appendix a.5, showing that retrievalqa is extremely hard for all models without access to external knowledge. we present detailed data statistics in ## improving adaptive rag prompting this section presents an improved model-based arag method and evaluates its effectiveness. ## conclusion this paper presents a new dataset retrievalqa to assess adaptive rag for short-form open-domain qa. we find vanilla prompting is insufficient in guiding llms to make reliable retrieval decisions. as an initial attempt, we propose ta-are, a simple yet effective method to help llms assess the necessity of retrieval, obviating the need for calibration or additional training.