Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models (2024)

Abstract: While large language models (LMs) demonstrate remarkable performance, they encounter challenges in providing accurate responses when queried for information beyond their pre-trained memorization. Although augmenting them with relevant external information can mitigate these issues, failure to consider the necessity of retrieval may adversely affect overall performance. Previous research has primarily focused on examining how entities influence retrieval models and knowledge recall in LMs, leaving other aspects relatively unexplored. In this work, our goal is to offer a more detailed, fact-centric analysis by exploring the effects of combinations of entities and relations. To facilitate this, we construct a new question answering (QA) dataset called WiTQA (Wikipedia Triple Question Answers). This dataset includes questions about entities and relations of various popularity levels, each accompanied by a supporting passage. Our extensive experiments with diverse LMs and retrievers reveal when retrieval does not consistently enhance LMs from the viewpoints of fact-centric popularity. Confirming earlier findings, we observe that larger LMs excel in recalling popular facts. However, they notably encounter difficulty with infrequent entity-relation pairs compared to retrievers. Interestingly, they can effectively retain popular relations of less common entities. We demonstrate the efficacy of our finer-grained metric and insights through an adaptive retrieval system that selectively employs retrieval and recall based on the frequencies of entities and relations in the question.

## introduction large language models (lms) @xcite have exhibited impressive capabilities owing to their ability to retrieve knowledge memorized during pre-training @xcite @xcite . however, despite the increase in model size and complexity, lms remain susceptible to factual inaccuracies in knowledge-intensive tasks such as open domain question answering and natural language generation, which demand access to a broader spectrum of information @xcite @xcite . retrieval-augmented language models (ralms) @xcite @xcite have emerged as a promising solution for mitigating factual errors by incorporating relevant external information. nevertheless, recent studies suggest that ralms are not a universal solution @xcite . indiscriminately augmenting lms with irrelevant passages can override potentially correct knowledge already possessed by the lm, resulting in incorrect responses (illustrated in determining when to recall and when to retrieve external information thus requires a thorough investigation of the following questions: while previous research has investigated factors influencing memorization in lms as well as performance of retrievers, they have a few limitations: a) they solely focus on entities @xcite , whereas real-world information comprises both entities and relations . b) they primarily focus exclusively on either retrievers or in this work, we aim to provide a finer-grained (fact-centric) analysis by investigating the impact of combinations of entities and relations on the performance of ralms. we focus on question answering (qa) task, where we analyze 10 lms of varying sizes with 5 different retrieval settings. to facilitate this, we require a qa benchmark that not only provides valid supporting passages for each qa pair but also integrates indicators for memorization in lms. additionally, the benchmark should encompass entities and relations of varying popularity. however, existing benchmarks such as popqa @xcite and entityquestions @xcite are not suitable for this purpose as they are entity-centric and target long-tailed information. to facilitate fact-centric analysis, we develop a novel dataset called witqa @xcite . we sample triples extracted from wikipedia, considering the popularity of entities and entity-relation combinations. we then generate qa pairs for each triple, ensuring that each example in witqa is accompanied by supporting passages and popularity scores for the question entity-relation pair. our investigation of ralms zero-shot performance on the proposed benchmark dataset (witqa) yields the following key findings: our findings reveal when retrieval does not assist lms through the lens of fact-level popularity. leveraging this insight, we propose a selective memory integration, which selectively employs retrieval augmentation and lms' memory based on the frequencies of entities and relations. our experiments demonstrate that this approach can enhance qa performance by up to 10.1%. ## the witqa dataset the witqa dataset is meticulously constructed with the underlying assumption that lms are likely to recall facts frequently mentioned in their pretraining corpus @xcite . we curate question-answer pairs annotated with the frequency of mentions of entities and their relationships within wikipedia-a predominant source of pre-training for lms-along with their supporting passages. this enables us to explore the correlation between popularity in the pretraining corpus and the performance of lms and retrieval models individually. furthermore, when operating in tandem, it facilitates the analysis of whether ralm's errors stem from lm's reasoning or primarily arise from retrieval errors. ## experiments: recall or retrieve we evaluate 10 language models of varying sizes augmented with four different retrieval methods to quantify the recall capability of lms and the performance of retrievers in isolation and jointly and share the insights. ## further related work several recent studies @xcite @xcite have focused on enhancing the robustness of ralms by assessing the usefulness and relevance of retrieved passages to questions. @xcite has proposed the chain-of-note framework, which systematically evaluates the relevance and accuracy of retrieved passages, thereby improving the noise robustness of ralms. @xcite has addressed query rewriting to improve retriever performance by using lms or pre-trained lms that are fine-tuned for a suitable query rewriter. @xcite has introduced an approach that leverages two sources of information: retrieved passages and lm-generated passages. this approach is designed on the hypothesis that answers corroborated by both sources have a higher likelihood of being accurate. these studies rely on model-centric approaches for assessing document relevance to queries, refining queries, and integrating retrieved and generated passages. thus, they overlook the importance of data-centric question popularity as an indicator for deciding when to retrieve and augment information. in contrast, our study leverages question popularity metrics derived from wikipedia, offering insights that are complementary and distinct from these existing methodologies. ## conclusion we introduced witqa, a novel qa dataset comprising of qa pairs associated with supporting passages. this dataset enables us to assess the performance of lms with respect to question popularity and retrieval-augmentation. we conducted extensive experiments investigating the zero-shot performance of 10 lms with four different retrieval methods. our findings reveal several key insights: 1) the ability to recall factual knowledge is influenced by the model's size, with even the gpt series facing challenges when addressing minor facts; 2) small ralms demonstrate robust qa performance when provided with supporting passages, suggesting that errors in ralms are primarily due to retrieval errors; and 3) retrievers exhibit greater robustness for long-tail information of long-tail entities compared to the recall capability of lms.