Generation-Augmented Retrieval for Open-Domain Question Answering (2021)

Abstract: We propose Generation-Augmented Retrieval (GAR) for answering open-domain questions, which augments a query through text generation of heuristically discovered relevant contexts without external resources as supervision. We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. Moreover, as sparse and dense representations are often complementary, GAR can be easily combined with DPR to achieve even better performance. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader, and consistently outperforms other retrieval methods when the same generative reader is used.

## introduction open-domain question answering (openqa) aims to answer factoid questions without a pre-specified domain and has numerous real-world applications. in openqa, a large collection of documents (e.g., wikipedia) are often used to seek information pertaining to the questions. one of the most common approaches uses a retriever-reader architecture @xcite , which first retrieves a small subset of documents using the question as the query and then reads the retrieved documents to extract (or generate) an answer. the retriever is crucial as it is infeasible to examine every piece of information in the entire document collection (e.g., millions of wikipedia passages) and the retrieval accuracy bounds the performance of the (extractive) reader. early openqa systems @xcite use classic retrieval methods such as tf-idf and bm25 with sparse representations. sparse methods are lightweight and efficient, but unable to perform semantic matching and fail to retrieve relevant passages without lexical overlap. more recently, methods based on dense representations @xcite learn to embed queries and passages into a latent vector space, in which text similarity beyond lexical overlap can be measured. dense retrieval methods can retrieve semantically relevant but lexically different passages and often achieve better performance than sparse methods. however, the dense models are more computationally expensive and suffer from information loss as they condense the entire text sequence into a fixed-size vector that does not guarantee exact matching @xcite . there have been some recent studies on query reformulation with text generation for other retrieval tasks, which, for example, rewrite the queries to context-independent @xcite @xcite or well-formed @xcite ones. however, these methods require either task-specific data (e.g., conversational contexts, ill-formed queries) or external resources such as paraphrase data @xcite ) that cannot or do not transfer well to openqa. also, some rely on timeconsuming training process like reinforcement learning (rl) @xcite @xcite ) that is not efficient enough for openqa (more discussions in sec. 2). in this paper, we propose generation-augmented retrieval (gar), which augments a query through text generation of a pre-trained language model (plm). different from prior studies that reformulate queries, gar does not require external resources or downstream feedback via rl as supervision, because it does not rewrite the query but expands it with heuristically discov-ered relevant contexts, which are fetched from plms and provide richer background information ( intuitively, the generated contexts explicitly express the search intent not presented in the original query. as a result, gar with sparse representations achieves comparable or even better performance than state-of-the-art approaches @xcite with dense representations of the original queries, while being more lightweight and efficient in terms of both training and inference (including the cost of the generation model) (sec. 6.4). specifically, we expand the query (question) by adding relevant contexts as follows. we conduct seq2seq learning with the question as the input and various freely accessible in-domain contexts as the output such as the answer, the sentence where the answer belongs to, and the title of a passage that contains the answer. we then append the generated contexts to the question as the generationaugmented query for retrieval. we demonstrate that using multiple contexts from diverse generation targets is beneficial as fusing the retrieval results of different generation-augmented queries consistently yields better retrieval accuracy. we conduct extensive experiments on the natural questions (nq) @xcite and triviaqa (trivia) @xcite datasets. the results reveal four major advantages of gar: (1) gar, combined with bm25, achieves significant gains over the same bm25 model that uses the original queries or existing unsupervised query expansion (qe) methods. (2) gar with sparse representations (bm25) achieves comparable or even better performance than the current state-of-the-art retrieval methods, such as dpr @xcite , that use dense representations. (3) since gar uses sparse representations to measure lexical overlap foot_0 , it is complementary to dense representations: by fusing the retrieval results of gar and dpr, we obtain consistently better performance than either method used individually. (4) gar outperforms dpr in the end-to-end qa performance (em) when the same extractive reader is used: @xmath0.8 (43.8 when combining with dpr) on nq and 62.7 on trivia, creating new state-ofthe-art results for extractive openqa. gar also outperforms other retrieval methods under the generative setup when the same generative reader is used: @xmath1.1 (45.3 when combining with dpr) on nq and 62.2 on trivia. contributions. (1) we propose generation-augmented retrieval (gar), which augments queries with heuristically discovered relevant contexts through text generation without external supervision or time-consuming downstream feedback. (2) we show that using generation-augmented queries achieves significantly better retrieval and qa results than using the original queries or existing unsupervised qe methods. (3) we show that gar, combined with a simple bm25 model, achieves new state-of-the-art performance on two benchmark datasets in extractive openqa and competitive results in the generative setting. ## openqa with gar to further verify the effectiveness of gar, we equip it with both extractive and generative readers for end-to-end qa evaluation. we follow the reader design of the major baselines for a fair comparison, while virtually any existing qa reader can be used with gar. ## experiment results we evaluate the effectiveness of gar in three stages: generation of query contexts (sec. 6.1), retrieval of relevant passages (sec. 6.2), and passage reading for openqa (sec. 6.3). ablation studies are mostly shown on the nq dataset to understand the drawbacks of gar since it achieves question: when did bat out of hell get released? answer: september 1977 {september 1977} sentence: bat out of hell is the second studio album and the major -label debut by american rock singer meat loaf ... released in september 1977 on cleveland international / epic records. {the album was released in september 1977 on cleveland international / epic records.} title: bat out of hell {bat out of hell} question: who sings does he love me with reba? answer: brooks & dunn {linda davis} sentence: linda kaye davis ( born november 26, 1962 ) is an american country music singer. {" does he love you " is a song written by sandy knox and billy stritch, and recorded as a duet by american country music artists reba mcentire and linda davis.} title: does he love me [sep] does he love me (reba mcentire song) [sep] i do (reba mcentire album) {linda davis [sep] greatest hits volume two (reba mcentire album) [sep] does he love you} question: what is the name of wonder womans mother? answer: mother magda {queen hippolyta} sentence: in the amazonian myths, she is the daughter of the amazon queen sifrat and the male dwarf shuri, and is the mother of wonder woman. {wonder woman's origin story relates that she was sculpted from clay by her mother queen hippolyta and given life by aphrodite.} title: wonder woman [sep] diana prince [sep] wonder woman (2011 tv pilot) {wonder woman [sep] orana (comics) [sep] wonder woman (tv series)} ## conclusion in this work, we propose generation-augmented retrieval and demonstrate that the relevant contexts generated by plms without external supervision can significantly enrich query semantics and improve retrieval accuracy. remarkably, gar with sparse representations performs similarly or better than state-of-the-art methods based on the dense representations of the original queries. gar can also be easily combined with dense representations to produce even better results. furthermore, gar achieves state-of-the-art end-to-end performance on extractive openqa and competitive performance under the generative setup.