Context Generation Improves Open Domain Question Answering (2023)

Abstract: Closed-book question answering (QA) requires a model to directly answer an open-domain question without access to any external knowledge. Prior work on closed-book QA either directly finetunes or prompts a pretrained language model (LM) to leverage the stored knowledge. However, they do not fully exploit the parameterized knowledge. To address this inefficiency, we propose a two-stage, closed-book QA framework which employs a coarse-to-fine approach to extract the relevant knowledge and answer a question. We first generate a related context for a given question by prompting a pretrained LM. We then prompt the same LM to generate an answer using the generated context and the question. Additionally, we marginalize over the generated contexts to improve the accuracies and reduce context uncertainty. Experimental results on three QA benchmarks show that our method significantly outperforms previous closed-book QA methods. For example on TriviaQA, our method improves exact match accuracy from 55.3% to 68.6%, and is on par with open-book QA methods (68.6% vs. 68.0%). Our results show that our new methodology is able to better exploit the stored knowledge in pretrained LMs without adding extra learnable parameters or needing finetuning, and paves the way for hybrid models that integrate pretrained LMs with external knowledge.

## introduction open-domain question answering (odqa) produces an answer to a given question in the form of natural language, and the task has been extensively studied in recent years. significant progress on odqa has been made by developing the openbook qa methods @xcite @xcite ; figure 1 : an example illustrating our two-stage, cgap framework. cgap generates more accurate answer (e.g. richard marx) compared to standard few-shot prompting (e.g. ross bagdasarian). @xcite that explicitly exploit external knowledge corpus via dense retrieval techniques like dpr @xcite . however, learning a good retriever requires substantial resources, such as a large number of domain-specific pairs of question and contexts in the knowledge corpus @xcite , or intensive compute resources @xcite . in addition, as the size of the knowledge corpus increases, it becomes harder to retrieve accurate contexts due to the high dimensionality of the search space @xcite . another class of models, known as closed-book question answering (cbqa), were recently proposed @xcite . cbqa tries to directly answer the open-domain questions without accessing any external knowledge sources, and instead leverages the parametric knowledge stored in the pretrained language models (lms) @xcite @xcite . however, even with larger lms, the closed-book methods are not competitive with the open-book methods in term of accuracy @xcite . while it has been shown that large pretrained lms store an abundant amount of knowledge @xcite , we hypothesize the accuracy gaps are largely because the way of exploiting the parameterized knowledge are not sophisticated enough. prior works on cbqa either finetune pretrained lm models on the entire qa datasets @xcite , or they directly prompt those models using several few-shot qa pairs @xcite . on the contrary, open-book models use a two-stage pipeline. they first retrieve relevant contexts from external corpus, then they extract the answer based on the retrieved contexts. therefore, to better exploit the parameterized knowledge in pretrained lms and bridge the large accuracy gaps between the closed-book and openbook methods, we propose a coarse-to-fine, twostage method for cbqa task. the main idea is to leverage generated contexts as an intermediate bridge between the huge amount of parameterized knowledge stored in the lm and the answer that lies within this knowledge. to the best of our knowledge, no previous work has been conducted on generating context from large pretrained lms for cbqa and leveraging them to predict answer. our proposed framework cgap consists of two stages. it first performs context generation relevant to a given question by prompting a pretrained lm. it then prompts the same lm for answer prediction using the generated context and the question. in order to improve the accuracies and to reduce context uncertainties, we generate multiple contexts for each question and predict the final answer by majority voting. this step does not increase the inference cost as we generate the contexts in parallel by batching in a single inference call. figure 1 illustrates how our two stage prompting and majority voting works. for the input question, cgap generates 3 contexts and 3 predicted answers at the two stages respectively, and choose the most voted answer as the final answer. note that we do not finetune the large pretrained lms for context generation or answer prediction. this facilitates our approach to take advantage of large lms such as gpt-3 @xcite , palm @xcite or megatron-turing nlg 530b @xcite , which are only available through apis. we conduct in-depth experimental studies on three open-domain qa benchmarks, natural questions @xcite , webquestions @xcite , and triviaqa @xcite , and demonstrate significant improvements by our two stage prompting method. our contributions are summarized as follows: to the best of our knowledge, we are the first to leverage generated context from large pretrained lms for open-domain question answering. ## methodology our proposed context geration and answer prediction (cgap) framework is illustrated in figure 2. cgap consists of two stages. first, it generates relevant context to a given question by prompting a large pretrained lm. in the second stage, it predicts an answer using the generated context and the question by prompting the same lm. to accurately predict the answer, we generate multiple contexts. we run each of the two stages multiple times in parallel in batch for the same question, generating different contexts for each, and use majority voting to select the final answer. formally, for our task we have a question q to be answered, and a support repository @xmath0, q 1 , a 1 ), . . . , (c n , q n , a n )} that consists of tuples of question q i and answer a i pairs with mapping to the context c i . in our experiments, we use the training sets of the corresponding datasets as d. ## results and ablation studies we now show our main results as well as ablations to further analyze the effectiveness of our approach. ## related works open-domain qa is the task of answering general-domain questions @xcite , in which the evidence is usually not given. models that explicitly exploit an external corpus are referred as open-book models @xcite . they typically index the corpus and then retrieveand-read to extract the answer span from documents @xcite @xcite @xcite . another recently proposed class of methods is closed-book qa models. @xcite ; @xcite finetune pretrained lms such as t5 @xcite or bart @xcite with qa pairs without access to any external knowledge or context. ## conclusion we propose a simple yet effective framework named cgap for open-domain qa. cgap performs context generation followed by answer prediction via two-stage prompting using large pretrained lms. it does not rely on external knowledge sources, and does not need finetuning or add extra learnable parameters. to the best of our knowledge, we are the first to leverage generated context from large pretrained lms for open-domain qa. experimental results on three qa benchmarks show that our method significantly outperforms previous closed-book qa methods and is par with openbook methods. we demonstrate our method up to 530b parameter models and showcase that larger models boost the accuracy by huge margins.