Retrieval-based Language Models and Applications (2023)

Abstract: Retrieval-based language models (LMs) have shown impressive performance on diverse NLP tasks. In this tutorial, we will provide a comprehensive and coherent overview of recent advances in retrieval-based LMs. We will start by providing preliminaries covering the foundation of LMs (e.g., masked LMs, autoregressive LMs) and retrieval systems (e.g., nearest-neighbor search). We will then detail recent progress in retrieval-based models, focusing on their model architectures and learning approaches. Finally, we will show how retrieval-based LMs are adapted to downstream applications, and extended to multilingual and multi-modal settings. Finally, we will use an exercise to showcase the effectiveness of retrieval-based LMs.

## description language models (lms) such as gpt-3 @xcite and palm @xcite have shown impressive abilities in a range of natural language processing (nlp) tasks. however, relying solely on their parameters to encode a wealth of world knowledge requires a prohibitively large number of parameters and hence massive compute, and they often struggle to learn long-rail knowledge @xcite @xcite . moreover, these parametric lms are fundamentally incapable of adapting over time @xcite @xcite , often hallucinate @xcite , and may leak private data from the training corpus @xcite . to overcome these limitations, there has been growing interest in retrieval-based lms @xcite @xcite @xcite , which incorporate a non-parametric datastore (e.g., text chunks from an external corpus) with their parametric counterparts. retrieval-based lms can outperform lms without retrieval by a large margin with much fewer parameters @xcite , can update their knowledge by replacing their retrieval corpora @xcite , and provide citations for users to easily verify and evaluate the predictions @xcite . previously, retrieval and lms have been studied mostly separately, and only recently researchers have integrated them and built systems in which retrieval and lms interact more organically, and a number of retrieval-based lms have been proposed due to growing interest. they differ in their neural architectures (e.g., the granularity of retrieval units, how to integrate retrieved information), learning algorithms, and different uses in downstream applications. in this tutorial, we aim to provide a comprehensive and coherent overview of recent advances in retrieval-based lms. we will start by first providing preliminaries covering the foundations of lm (e.g., masked lms, autoregressive lms) and retrieval systems (e.g., nearest-neighbor search methods widely used in neural retrieval systems; @xcite ). we will then focus on recent progress in architectures, learning approaches, and applications of retrieval-based lms. a taxonomy of architectures we introduce a taxonomy of architectures of retrieval-based lms based on a variety of dimensions. retrieval-based lms can be categorized by the granularity of retrieved units stored in the datastore: either 1) a chunk of text @xcite , or 2) a token @xcite @xcite , or 3) an entity mention @xcite . we also plan to cover techniques for refining data stores and improving similarity search @xcite . at the same time, retrieval-base lms can be categorized based on how the retrieved information is integrated with the parametric encoder: 1) whether retrieved components are concatenated with the original input text @xcite @xcite , 2) whether the retrieved components are latent and integrated into the intermediate layers of transformers @xcite @xcite , or 3) distribution of tokens from the retrieved components and the lms are interpolated @xcite @xcite . scalable learning algorithms then, we discuss the training approaches of retrieval-based lms. since a retrieval datastore is typically very large, how to train retrieval-based lms effectively and efficiently remains challenging. we first discuss pipelined approaches that train retrieval components and lms separately, either through large-scale pre-training @xcite or multitask instruction tuning @xcite . several other works train retrieval-based lms with a fixed retrieval module @xcite . we then discuss joint training under reasonable resource requirements: either through in-batch approximations to a full datastore, or updating the datastore with updated parameters asynchronously. the former uses fractions of the full corpus that are carefully designed during joint training @xcite @xcite . the latter, on the other hand, aims to use full corpus during training with asynchronous index update for every certain time steps @xcite . adaption to downstream tasks after discussing the basic building blocks of retrieval-based lms, we show how retrieval-based lms are adapted to downstream applications. we first briefly summarize the two approaches to adapt a model to a new task: zero-shot or few-shot prompting without any parameter updates @xcite , and fine-tuning on target task data @xcite ). we then discuss methods designed to build more powerful retrieval-based lms for certain downstream tasks, such as dialogue @xcite , semantic parsing @xcite , and machine translation @xcite . up to this point, our tutorial has mainly focused on retrieving and integrating english plain text. at this end, we will cover recent extensions of retrieval-based lms beyond english text, including multilingual @xcite , multimodal @xcite and code (parvez et al., 2021) retrieval. these works often extend dense retrieval models to enable retrieval between heterogeneous input spaces (e.g., cross-lingual, cross-modal) and have shown that referring retrieved knowledge leads to knowledgeintensive generation. finally, we will use an exercise to showcase the effectiveness of retrieval-based lms. we conclude our tutorial by discussing several important questions and future directions, including (1) how we can further improve the scalability of retrievalbased lms without sacrificing performance, (2) when retrieval-based lms are particularly useful in the era of rapidly evolving lms, and ( 3 ) what is necessary to enable applications of retrieval-based lms for more diverse domains. length this is a 3-hour tutorial. target audience the tutorial will be accessible to anyone who has a basic knowledge of machine learning and natural language processing. we think the topic will be of interest to both nlp researchers/students in academia and nlp practitioners in the industry.