Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models (2023)

Abstract: Augmenting language models with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low. Retrieval-augmented models commonly rely on a semantic retrieval mechanism based on the similarity between dense representations of the query chunk and potential neighbors. In this paper, we study the state-of-the-art Retro model and observe that its performance gain is better explained by surface-level similarities, such as token overlap. Inspired by this, we replace the semantic retrieval in Retro with a surface-level method based on BM25, obtaining a significant reduction in perplexity. As full BM25 retrieval can be computationally costly for large datasets, we also apply it in a re-ranking scenario, gaining part of the perplexity reduction with minimal computational overhead.

## introduction the introduction of the transformer architecture @xcite has led to a performance boost in language modeling (see, e.g., @xcite , but also to a steep increase of computational cost, as the number of parameters and data points is constantly growing. in reaction to this development, there has recently been a surge in work on retrieval-augmented language models @xcite , which shows that enabling models to retrieve context from large corpora results in lower perplexity and better accuracy in downstream tasks such as question answering, while at the same time using considerably fewer parameters. in this paper, we specifically focus on the retrieval-enhanced transformer architecture (retro; @xcite . by augmenting a language model with a retrieval mechanism, retro, like similar architectures, tries to decouple memorization of the training data from the additional generalization that * correspondence to ehsan.doostmohammadi@liu.se. ## method we experiment with retro @xcite as a state-of-the-art retrieval-augmented language model. ## a study on correlations we experiment with two settings: retro[on], the language model with retrieval enabled, and retro @xcite , where there are no chunk crossattention layers and therefore no retrieval, leaving us with a decoder-only language model. as shown by @xcite , the retro[on] model performs better when it can exploit an overlap between the generated text and the retrieved neighbor. this is more apparent in text categories with higher token overlap, such as github. the studies in the retro paper also show that allowing more overlap when deduplicating the data results in a lower bits-per-byte (bpb foot_1 ). @xcite take this further to show even minimal overlap results in significant loss reduction, demonstrating the large extent retro relies on surface-level similarities. these findings lead us to hypothesize that having a retrieval method that can find the highest overlapping neighbors will yield lower perplexity (ppl). because bert, st and similar deep representations of sentences do not always capture surfacelevel similarities, we set out to investigate where performance gains come from. to this end, we measure how the ppl difference (∆ppl) between @xcite and @xcite for the current chunk (c u , @xmath0, and (2) unigram token overlap, based on t5 tokenization, between c u and ret(c u-1 ). the results, reported in ## changing the retrieval method as the results from the previous section show a stronger correlation between performance gain and surface-level similarity than st similarity, we experiment with a retrieval method based on bm25. ## related work augmenting language models with mechanisms that help them incorporate larger contexts has been approached extensively in different forms, such as @xcite 's retrieve-and-edit approach to reduce the ppl in language generation, and @xcite that make use of lexical overlap to improve the performance in question answering. while retrieval-augmentation has been used with different objectives in mind, such as language modeling @xcite and machine translation @xcite , question answering has been the application to attract the most interest @xcite @xcite ). an extensive study was performed by @xcite , showing that while we get performance gains using retrieval augmentation, training the retrieval part of the model would yield even more benefits. retro @xcite , on the other hand, aims at scaling such language models and therefore opts for keeping the retriever frozen, showing substantial ppl reduction with increasing either the number of language model parameters or the size of retrieval set. among the more recent work, xu et al. (2023) found that training using approximate neighbors resulted in a 2.6% decrease in perplexity. this suggests that non-exact neighbors may have a regularization effect, leading to improved generalization ability. additionally, @xcite report a drop in perplexity using bm25 over bert retrieval using in-context retrieval-augmented language models. ## conclusions and future work in this paper, we study the source of performance gains in retro, which could be generalized to similar retrieval-augmented language models. after observing that the ppl drop correlates more strongly with surface-level overlap between the query and the retrieved text, we replace the retrieval method with bm25, and observe a significant drop in ppl, which confirms us in the findings of the correlation study. this is also an interesting insight as to how these models work, which could be lever-aged for performance gain in tasks like question answering where model relies on retrieving facts. in the end, we also conduct an analysis to find out how much bm25 neighbors overlap with those retrieved using st. the results show that while faiss is able to find some of the neighbors with high token overlap, the majority of them remain unretrieved. this is however, enough to gain part of the loss reduction achieved with a pure bm25 retrieval system. the proposed methods could also be used during training. by retrieving more overlapping neighbors during training, the process of guiding the model to use retrieved neighbors for language modeling could be done more efficiently. this is particularly relevant when augmenting an already trained language model with a retrieval mechanism. as reported by @xcite , retrieval augmentation results in a larger drop in bpb as the number of model parameters and the size of retrieval data grow. this calls for more efficient methods based on surface-level similarities if we wish to exploit this potential. furthermore, the retrieval system in retro is based on semantic retrieval, the model seems to rely more on surface-level similarities. this could affect the generalizability capabilities of such models, which necessitates further investigations. lastly, we only evaluate our modified retro model on language modeling. it would be interesting to know the impacts of bm25 retrieval on downstream tasks where retrieval is of use.