In-Context Retrieval-Augmented Language Models (2023)

Abstract: Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.1

## introduction recent advances in language models (lms) have dramatically increased the usefulness of machine-generated text across a wide range of use-cases and domains @xcite . however, the mainstream paradigm of generating text with lms bears inherent limitations in access to external knowledge. first, lms are not coupled with any source attribution, and must be trained in order to incorporate up-to-date information that was not seen during training. more importantly, they tend to produce factual inaccuracies and errors @xcite @xcite . this problem is present in any lm generation scenario, and is exacerbated when generation is made in uncommon domains or private data. a promising approach for addressing the above is retrieval-augmented language modeling (ralm), grounding the lm during generation by conditioning on relevant documents retrieved from an external knowledge source. ralm systems include two high level components: (i) document selection, selecting the set of documents upon which to condition; and (ii) document reading, determining how to incorporate the selected documents into the lm generation process. leading ralm systems introduced recently tend to be focused on altering the language model architecture @xcite @xcite @xcite . notably, @xcite introduced retro, featuring document reading via nontrivial modifications that require further training to the lm architecture, while using an off-the-shelf frozen bert retriever for document selection. although the paper's experimental findings showed impressive performance gains, the need for changes in architecture and dedicated retraining has hindered the wide adoption of such models. in this paper, we show that a very simple document reading mechanism can have a large impact, and that substantial gains can also be made by adapting the document selection mechanism to the task of language modeling. thus, we show that many of the benefits of ralm can be achieved while working with off-the-shelf lms, even via api access. specifically, we consider a simple but powerful ralm framework, dubbed in-context ralm (presented in section 3), which employs a zero-effort document reading mechanism: we simply prepend the selected documents to the lm's input text (figure 1 ). section 4 describes our experimental setup. to show the wide applicability of our framework, we performed lm experiments on a suite of five diverse corpora: wikitext-103 @xcite , realnews @xcite , and three datasets from the pile @xcite : arxiv, stack exchange, and freelaw. we use open-source lms ranging from 110m to 66b parameters (from the gpt-2, gpt-neo, opt, and llama model families). in section 5 we evaluate the application of off-the-shelf retrievers to our framework. in this minimal-effort setting, we found that in-context ralm led to lm performance gains equivalent to increasing the lm's number of parameters by 2-3× across all of the text corpora we examined. in section 6 we investigate methods for adapting document ranking to the lm task, a relatively under-explored ralm degree of freedom. our adaptation methods range from using a small lm to perform zero-shot ranking of the retrieved documents, up to training a dedicated bidirectional reranker by employing self-supervision from the lm signal. these methods lead to further gains in the lm task corresponding to an additional size increase of 2× in the lm architecture. as a concrete example of the gains, a 345m parameter gpt-2 enhanced by in-context ralm outperforms a 762m parameter gpt-2 when employing an off-the-shelf bm25 retriever @xcite , and outperforms a 1.5b parameter gpt-2 when employing our trained lm-oriented reranker (see figure 2 ). for large model sizes, our method is even more effective: in-context ralm with an off-the-shelf retriever improved the performance of a 6.7b parameter opt model to match that of a 66b parameter parameter opt model (see figure 4 ). provides large language modeling gains on the test set of wikitext-103, without modifying the lm. adapting the use of a bm25 retriever @xcite to the lm task ( §5) yields significant gains, and choosing the grounding documents via our new class of predictive rerankers ( §6) provides a further boost. see in section 7 we demonstrate the applicability of in-context ralm to downstream open-domain questions answering (odqa) tasks. in a concurrent work, @xcite also suggest to augment off-the-shelf lms with retrieved texts by prepending them to the input. their results are based on training a dedicated retriever for language modeling. in contrast, we focus on the gains achievable in using off-the-shelf retrievers for this task. we show strong gains of this simpler setting by investigating: (1) which off-the-shelf retriever is best suited for language modeling, (2) the frequency of retrieval operations, and (3) the optimal query length. in addition, we boost the off-the-shelf retrieval performance by introducing two reranking methods that demonstrate further gains in perplexity. we believe that in-context ralm can play two important roles in making ralm systems more powerful and more prevalent. first, given its simple reading mechanism, in-context ralm can serve as a clean probe for developing document retrieval methods that are specialized for the lm task. these in turn can be used to improve both in-context ralm and other more elaborate ralm methods that currently leverage general purpose retrievers. second, due to its compatibility with off-the-shelf lms, in-context ralm can help drive wider deployment of ralm systems. ## experimental details we now describe our experimental setup, including all models we use and their implementation details. ## the effectiveness of in-context ralm with off-the-shelf retrievers we now empirically show that despite its simple document reading mechanism, in-context ralm leads to substantial lm gains across our diverse evaluation suite. we begin in this section by investigating the effectiveness of off-the-shelf retrievers for in-context ralm; we go on in §6 to show that further lm gains can be made by tailoring document ranking functions to the lm task. the experiments in this section provided us with a recommended configuration for applying in-context ralm: applying a sparse bm25 retriever that receives ℓ = 32 query tokens and is applied as frequently as possible. practically, we retrieve every @xmath0. ## . 6 llama-33b -6 . 3 bm25, §5 6 . 1 ## in-context ralm for open-domain question answering so far, we evaluated our framework on language modeling benchmarks. to test its efficacy in additional scenarios, and specifically downstream tasks, we now turn to evaluate in-context ralm on open-domain question answering (odqa; @xcite . this experiment is intended to verify, in a controlled environment, that lms can leverage retrieved documents without fur-model retrieval nq triviaqa llama-7b -10.3 47.5 dpr 28.0 56.0 llama-13b -12.0 54.8 dpr 31.0 60.1 llama-33b -13.7 58.3 dpr 32.3 62.7 varying the number of documents to investigate the the effect of the number of documents shown to the model, we performed a minimal analysis on the development set of nq and triviaqa. figure 8 demonstrates that showing documents in-context significantly improves the model's performance. in addition, most of the gain can be obtained by using only two documents (or even a single one in some cases). results ## discussion retrieval from external sources has become a common practice in knowledge-intensive tasks (such as factual question answering, fact checking, and more; @xcite . in parallel, recent breakthroughs in lm generation capabilities has led to lms that can generate useful long texts. however, factual inaccuracies remain a common way in which machine-generated text can fall short, and lack of direct provenance makes it hard to trust machine generated text. this makes language modeling both a promising and an urgent new application area for knowledge grounding, and motivates promoting ralm approaches. prior research has already investigated ralm, of course, but it is not yet widely deployed. one likely reason is that existing approaches rely upon fine-tuning the lm, which is typically difficult and costly, and is even impossible for lms accessible only via an api. this paper presented the framework of in-context ralm, enabling frozen, off-the-shelf lms to benefit from retrieval. we demonstrated that substantial performance gains can be achieved by using general purpose retrievers, and showed that additional gains can be achieved by tailoring the document selection to the lm setting. a recent work by @xcite demonstrates that in-context ralm is indeed able to improve the factuality of large lms. several directions for further improvement remain for future work. first, this paper considers only the case of prepending a single external document to the context; adding more documents could drive further gains (for example, using the framework of @xcite . second, we retrieved documents every fixed interval of s tokens, but see potential for large latency and cost gains by retrieving more sparsely, such as only when a specialized model predicts that retrieval is needed. we release the code used in this work, for the community to use and improve over. we hope it will drive further research of ralm, which will enable its wider adoption.