Retrieval-Augmented Knowledge Integration into Language Models: A Survey (2024)

Abstract: This survey analyses how external knowledge can be integrated into language models in the context of retrieval-augmentation.The main goal of this work is to give an overview of: (1) Which external knowledge can be augmented? (2) Given a knowledge source, how to retrieve from it and then integrate the retrieved knowledge? To achieve this, we define and give a mathematical formulation of retrieval-augmented knowledge integration (RAKI). We discuss retrieval and integration techniques separately in detail, for each of the following knowledge formats: knowledge graph, tabular and natural language.

## introduction in natural language processing (nlp), external knowledge or information refers to information that is not explicitly present in the language model (lm) input yet helpful for lms to produce target output @xcite . traditional methods to integrate knowledge, especially those before large language models (llms) @xcite , include pre-training over a knowledge corpus @xcite @xcite , and fine-tuning in the domain that the knowledge is concerned with @xcite . despite improved performance of the resulting models @xcite , such methods typically require (re-)training on the whole (without filtering) knowledge. this is not efficient, as the ever-growing size of language models @xcite raises hardware and energy issues @xcite of applying these training-intensive methods originally proposed for smaller models. as an alternative to traditional pre-training and fine-tuning to integrate knowledge into llms, retrieval-augmented (ra) methods @xcite have become more and more popular in recent years. ra methods leverage pre-trained internal knowledge already parameterized in lms as well as retrieved external knowledge @xcite . in the setting of retrieval augmentation, lms access for instance only the most relevant, top-k retrieved items without seeing the entire external sources, thus enabling efficiency @xcite . previous works also demonstrate decoupling knowledge and language model can lead to better adaptability @xcite , straightforward knowledge edit @xcite and improved explainability @xcite . to track the research intersection of retrieving knowledge to augment lms, we study the topic of retrieval-augmented knowledge integration (raki) in this survey. in raki, the retrieval base is some specific external knowledge @xcite (e.g. a knowledge graph or a set of wikipedia articles), where the knowledge is typically written by experts and thus enjoys higher factuality than general texts. this survey is mainly based on recent @xcite @xcite @xcite @xcite publications (see appendix a.1, a.2 for more details of literature). inspired by @xcite , we categorize the published works in this line of research based on the format of knowledge source: knowledge graph, tabular and natural language. for each knowledge source, we start by introducing the source format using the annotations proposed in section 2. then, we discuss in detail the retrieval and integration techniques proposed in the reviewed methods. finally, we point out the challenges of raki and list some relevant work to deal with them. we would like to point out that this survey aims to focus on (pure) nlp and does not consider work on vision @xcite or audio @xcite . ## preliminaries in the following, we briefly introduce retrievalaugmented generation (rag) and then define and formulate retrieval-augmented knowledge integration (raki). retrieval-augmented generation is first proposed by @xcite , where world knowledge is retrieved from a vector index constructed over wikipedia articles and then sent to a seq2seq @xcite model for generation. more formally, given an input-output pair (x, y) from a generation task, retrieval-augmented generation aims to generate the target output y conditioned on the input x and an accessible document set z for reference @xcite . ## challenges & outlook here we summarize some challenges of retrievalaugmented knowledge integration techniques, followed by an outlook of the raki framework. ## conclusion this survey paper studies recent works that augment language models by retrieving external knowledge sources. we categorize research in retrievalaugmented knowledge integration (raki) into three sections, according to knowledge format: knowledge graph, tabular, and natural language. besides a comprehensive collection of knowledge retrieval and integration approaches, we also point out the limitations and challenges of current raki. we hope this survey could (1) help researchers who are looking for a technical-intensive overview and (2) encourage future work to improve current raki.