Retrieval-augmented Generation across Heterogeneous Knowledge (2022)

Abstract: Retrieval-augmented generation (RAG) methods have been receiving increasing attention from the NLP community and achieved state-of-the-art performance on many NLP downstream tasks. Compared with conventional pre-trained generation models, RAG methods have remarkable advantages such as easy knowledge acquisition, strong scalability, and low training cost. Although existing RAG models have been applied to various knowledge-intensive NLP tasks, such as open-domain QA and dialogue systems, most of the work has focused on retrieving unstructured text documents from Wikipedia. In this paper, I first elaborate on the current obstacles to retrieving knowledge from a single-source homogeneous corpus. Then, I demonstrate evidence from both existing literature and my experiments, and provide multiple solutions on retrieval-augmented generation methods across heterogeneous knowledge.

## introduction in recent years, large pre-trained language models (plms), such as t5 @xcite and gpt-3 @xcite , have revolutionized the field of natural language processing (nlp), achieving remarkable performance on various downstream tasks @xcite . these plms have learned a substantial amount of in-depth knowledge from the pre-training corpus @xcite , so they can predict the outputs on downstream tasks without access to any external memory or raw text, as a parameterized implicit knowledge base @xcite . the way of fine-tuning plms using only input-output pairs of target data is often referred to as close-book setting @xcite . while this development is exhilarating, such large-scale plms still suffer from the following drawbacks: (i) they are usually trained offline, making the model agnostic to the latest information, e.g., asking a chat-bot trained from 2011-2018 about covid-19 @xcite . (ii) they make predictions by only "looking up information" stored in its parameters, leading to inferior interpretability @xcite . (iii) they are mostly trained on general domain corpora, making them less effective on domain-specific tasks @xcite . (iv) their pre-training phase can be prohibitively expensive for academic research groups, limiting the model pre-training to only a few industry labs @xcite . the solution that seems obvious at first glance is to allow language models free access to open-world resources, such as encyclopedias and books. the way of augmenting the input of plms with external information is often referred to as open-book setting @xcite . a prominent method in the open-book setting is retrieval-augmented generation (rag) @xcite , a new learning paradigm that fuses plms and traditional ir techniques, which has achieved state-of-the-art performance in many knowledgeintensive nlp tasks @xcite . compared with large-scale plms counterparts, e.g., gpt-3, the rag model has some remarkable ad-vantages: (i) the knowledge is not implicitly stored in model parameters, but is explicitly acquired in a plug-and-play manner, leading to great scalability; (ii) instead of generating from scratch, the model generates outputs based on some retrieved references, which eases the difficulty of text generation. although the rag models have been widely used in the existing literature, most of the work has focused on retrieving unstructured text from general domain corpus, e.g., wikipedia. however, the performance is often limited by the coverage of only one certain knowledge. for example, only a finite portion of questions could be answered from wikipedia passages in many open-domain qa datasets, while the remaining could only rely on the input question because no supportive documents could be retrieved @xcite . in this paper, i first elaborate on the current obstacles to retrieving knowledge from a single-source homogeneous corpus. then, i demonstrate several pieces of evidence from both existing literature and my own experiments, and provide multiple potential solutions on retrieval-augmented generation methods across heterogeneous knowledge. overall, a general rag framework has two major components: (i) a document retriever and (ii) a text generator, as shown in figure 2 . the objective of the rag is to train a model to maximize the likelihood of y given x and z, in practice, z often contains millions of documents, rendering enumeration over z impossible. therefore, the first step of rag is to leverage a document retriever, e.g., dpr @xcite , to narrow down the search to a handful of relevant documents. the retriever takes x and z as input and yields relevance scores {s 1 , • • • , s k } of the top-k documents @xmath0, • • • , z (k) }. then, the second step of rag is to use a text generator, e.g., bart @xcite and t5 @xcite ., the beatles john lm encoder q: who was the drummer for the beatles? a: ringo starr ringo starr paul lm decoder retriever close-book (plms) open-book (rag) figure 2: compared with plms, rag models directly seeks knowledge (e.g., texts, tables and kgs) from external information sources to help answer questions.