When to Read Documents or QA History: On Unified and Selective Open-domain QA (2023)

Abstract: This paper studies the problem of open-domain question answering, with the aim of answering a diverse range of questions leveraging knowledge resources. Two types of sources, QA-pair and document corpora, have been actively leveraged with the following complementary strength. The former is highly precise when the paraphrase of given question q was seen and answered during training, often posed as a retrieval problem, while the latter generalizes better for unseen questions. A natural follow-up is thus leveraging both models, while a naive pipelining or integration approaches have failed to bring additional gains over either model alone. Our distinction is interpreting the problem as calibration, which estimates the confidence of predicted answers as an indicator to decide when to use a document or QA-pair corpus. The effectiveness of our method was validated on widely adopted benchmarks such as Natural Questions and TriviaQA.

## introduction open-domain question answering is a well-known task in natural language processing, aiming to answer factoid questions from an open set of domains. one commonly used approach for this task is the retrieve-then-read pipeline (also known as openbook qa) to retrieve relevant knowledge, then reason answers over the knowledge. given the wide range of topics that open-domain questions can cover, a key to a successful answering model is: to access and utilize diverse knowledge sources effectively. toward this goal, existing work can be categorized by the knowledge source used: in an effort to leverage complementary strengths of existing models, previous work has attempted to build a pipeline of individual models @xcite . however, their approach has not resulted in significant gains over using either model alone. in this paper, we propose a novel approach of leveraging the strengths of both document and qa pairs as contexts for a unified reader-based qa (or ur-qa). foot_0 figure 1 illustrates the distinction of our approach providing both knowledge to a unified reader as context. we retrieve a list of relevant qa-pairs (called as qa-history), then treat the few retrieved qa examples, as if it is a relevant document passage. meanwhile, the closest approach to use multiple knowledge sources is concatenating the multisources uniformly into a single decoder @xcite ), but we argue knowledge selection is critically missing. to motivate, figure 1 shows the qa-history, from which answer 'eric liddell' is explicitly identified, while it is more implicit in the document such that another name such as 'hugh hudson' is known to often confuse qa models. it is critical for the qa model to calibrate prediction quality as an indicator to decide when to use a document corpus or qa-history. toward the goal, we propose selective qa, where a more reliable answer among candidates can be identified through the calibration of the qa model. existing calibration @xcite @xcite has focused on the ability of models to "know when they don't know" and abstain from answering if they are uncertain. a naive approach would be simply prioritizing more confident predictions for answer selection. as a known measure of confidence, lm likelihood of generated tokens has been found to often miscalibrate @xcite , tending to prefer short outputs @xcite , or being biased towards more frequent words @xcite . we also observed similar issues in our setting, which we refer to as calibration overfitting -lm likelihoods are biased towards increasing confidence on both correct and wrong answers. our distinction is to overcome this limitation, by proposing two new objectives, for lowering confidence when the given context cannot answer the question (i.e., answerability), or when sampling uncertainty from decoder is high (i.e., sampling consistency). finally, building upon improved calibration, we carefully select among answer candidates inferred from document and qa-pairs. to summarize, we make the following contributions: a) we propose an open-domain qa model complementing document corpus with qa-pair corpus, and decide the selective usage between a document or qa-pair corpus through calibration. b) we evaluate our approach on natural questions @xcite and triviaqa @xcite , and our method can improve qa performance of existing models. c) we analyze how our method improves calibration and how it helps to select better answers. qr retrieving relevant qa pairs over a large collection of qa pairs is a more efficient alternative to doc-qa. @xcite build paq (for probably asked questions) -65m qa pairs: automatically-generated resources by using question generation techniques, and learn repaq retriever to efficiently extract the top-1 qa pair that is most similar to a target question, and uses its answer for answering the question. @xcite use answer aggregation heuristic to combine retrieved candidates of qa-pairs with candidates from other sources. @xcite also leverage retrieved qa-pairs, by fusing representations of the qa-pairs into language models. despite some gains, their generalizability for unseen questions is limited, compared to doc-qa, which motivates our approach of selectively combining with other knowledge. our distinction is to analyze and utilize the complementarity of doc-qa and qr, carefully selecting knowledge sources via calibration, while the previous work @xcite blindly concatenates all types of data into a single context. calibration has been studied for abstaining from answering when the model does not know. sources of calibration have been lm's likelihoods @xcite , classifier @xcite , and linguistic expressions @xcite @xcite . our distinction is exploring the use of calibration for selective qa, and overcoming the calibration overfitting we observed from existing methods, by proposing new likelihoods based on answerability and consistency. ## proposed method in this section, we formally describe doc-qa as backbones (section 3.1) and our unification baseline (section 3.2), followed by our proposed calibration for selective qa (section 3.3). ## experiment in our experiments, we first demonstrate that our proposed confidence scores effectively improve the calibration for question answering. we then examine how these scores contribute to an overall improvement in question answering performance. finally, we provide qualitative analysis to gain a deeper understanding and insight on our method. ## acknowledgement this work was supported by lg ai research. this work was partly supported by institute of information & communications technology planning & evaluation (iitp) grant funded by the korea government(msit) [no.2021-0-01343, artificial intelligence graduate school program (seoul national university)]. ## conclusion this paper studies the selective qa system leveraging both document and qa-pair corpus. for careful selection, we propose a novel and effective calibration method based on answerability and sampling consistency and leverage them for comparing and selecting two knowledge sources. on two benchmarks: nq and tqa, we empirically show our proposed methods outperform existing approaches for open-domain question answering tasks.