Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (2023)

Abstract: Augmenting pretrained language models with retrievers has shown promise in effectively solving common NLP problems, such as language modeling and question answering. In this paper, we evaluate the strengths and weaknesses of popular retriever-augmented language models, namely kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriever + Flan-T5, in reasoning over retrieved statements across different tasks. Our findings indicate that the simple similarity metric employed by retrievers is insufficient for retrieving all the necessary statements for reasoning. Additionally, the language models do not exhibit strong reasoning even when provided with only the required statements. Furthermore, when combined with imperfect retrievers, the performance of the language models becomes even worse, e.g., Flan-T5’s performance drops by 28.6% when retrieving 5 statements using Contriever. While larger language models improve performance, there is still a substantial room for enhancement. Our further analysis indicates that multihop retrieve-and-read is promising for large language models like GPT-3.5, but does not generalize to other language models like Flan-T5-xxl. The code is available at https://github.com/McGill-NLP/retriever-lm-reasoning.

## introduction parametric language models, such as decoder-only transformers (e.g. gpt), transformer encoder models (e.g. bert), and encoder-decoder transformers (e.g. t5), encode all necessary knowledge to solve a given task in their parameters and have demonstrated exceptional performance on many natural language tasks @xcite @xcite . nonparametric models improve these models further by augmenting them with knowledge retrievers @xcite @xcite or memory components (khandelwal et al., 1 the code is available at https://github.com/mcgillnlp/retriever-lm-reasoning . 5. a moon is located in space. 6. classifying is a kind of science process. ## background there has been a growing interest in investigating the reasoning abilities of parametric language models @xcite . in contrast, our work focuses on the reasoning abilities of retriever-augmented language models. this paper further studies the contributions of each of the models' components. phobos is named after the greek god of fear and panic. retriever language model moon phobos is a kind of [mask]. ## problem definition in the main experiments, we provide the models with a complete set of knowledge statements denoted as @xmath0, s 2 , . . . , s m } for each sample. in some cases, only a subset of these statements is essential for predicting the answer, referred to as gold statements. the primary objective of the models is to 1) retrieve a set of statements s @xmath1, r 2 , . . . , r k } ⊆ s which find necessary model # params model # params language models realm ∼270m fid ∼220m knn-lm ∼250m atlas ∼250m flan-t5-base ∼250m model size and multihop retrieve-and-read analysis flan-t5-small ∼80m flan-t5-xl ∼3b flan-t5-base ∼250m flan-t5-xxl ∼11b flan-t5-large ∼780m gpt-3.5 ∼175b and 2) effectively solve the target task through reasoning over the retrieved knowledge s r . a visualization of the task is illustrated in figure 2 . we control for model size wherever possible for comparable results among different models. we also study the effect of scaling the model size. ## conclusion this paper analyzes the reasoning abilities of retriever-augmented language models. we first evaluate popular retrieve-then-read models, including knn-lm, realm, dpr + fid, contriever + atlas, and contriever + flan-t5, through language modeling and question answering tasks. our experimental results indicate that retrievers fail to select all essential statements for reasoning when relying on the similarity between the query and statements. moreover, we observe that language models also struggle to reason over statements even when distracting information is absent. the performance deteriorates further when coupled with imperfect retrievers, as flan-t5's performance drops by 28.6% when retrieving 5 statements using contriever. furthermore, while larger language models show greater capabilities in reasoning, they still have a large room for improvement. additionally, our experiments on multihop retrieve-and-read show improvements on gpt-3.5, but these improvements do not generalize to other language models such as flan-t5-xxl. these findings present opportunities for future research to enhance the performance of retrievethen-read models by addressing the aforementioned limitations of retrievers or language models. moreover, the development of multihop retrieve-andread models holds promise for advancing reasoning tasks.