Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework (2022)

Abstract: Open-domain questions are likely to be open-ended and ambiguous, leading to multiple valid answers. Existing approaches typically adopt the rerank-then-read framework, where a reader reads top-ranking evidence to predict answers. According to our empirical analysis, this framework faces three problems: first, to leverage a large reader under a memory constraint, the reranker should select only a few relevant passages to cover diverse answers, while balancing relevance and diversity is non-trivial; second, the small reading budget prevents the reader from accessing valuable retrieved evidence filtered out by the reranker; third, when using a generative reader to predict answers all at once based on all selected evidence, whether a valid answer will be predicted also pathologically depends on evidence of some other valid answer(s). To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework, which separates the reasoning process of each answer so that we can make better use of retrieved evidence while also leveraging large models under the same memory constraint. Our framework achieves state-of-the-art results on two multi-answer datasets, and predicts significantly more gold answers than a rerank-then-read system that uses an oracle reranker.

## introduction open-domain question answering @xcite is a long-standing task where a question answering system goes through a largescale corpus to answer information-seeking questions. previous work typically assumes that there is only one well-defined answer for each question, or only requires systems to predict one correct answer, which largely simplifies the task. however, humans may lack sufficient knowledge or patience * *corresponding author: minlie huang. ## related work open-domain qa requires question answering systems to answer factoid questions by searching for evidence from a large-scale corpus such as wikipedia @xcite . the presence of many benchmarks has greatly promoted the development of this community, such as questions from real users like nq @xcite and webquestions @xcite , and trivia questions like quasar-t @xcite and triviaqa @xcite . all these benchmarks either assume that each question has only one answer with several alternative surface forms, or only require a system to predict one valid answer. a typical question answering system is a pipeline as follows: an efficient retriever retrieves relevant passages using sparse @xcite or dense @xcite @xcite representations; an optional passage reranker @xcite @xcite further narrows down the evidence; an extractive or generative reader @xcite ) predicts an answer conditioned on retrieved or top-ranking passages. nearly all previous work focused on locating passages covering at least one answer, or tried to predict one answer precisely. however, both @xcite and @xcite reported that there is genuine ambiguity in open-domain questions, resulting in multiple valid answers. to study the challenge of finding all valid answers for open-domain questions, @xcite proposed a new benchmark called ambigqa where questions are annotated with as many answers as possible. in this new task, the passage reranker becomes more vital in the rerank-then-read framework, particularly when only a few passages are allowed to feed a large reader due to memory constraints. this is because the reranker has to ensure that top-ranking passages are highly relevant and also cover diverse answers. despite state-of-the-art performance on ambigqa @xcite , according to our empirical analysis, applying the rerank-then-read framework to open-domain multi-answer qa faces the following problems: balancing relevance and diversity is nontrivial for the reranker due to unknown effect on the performance of the subsequent reader; when using a large reader under a fixed memory constraint, the small reading budget prevents it from making use of more retrieved evidence that is valuable but filtered out; when using a generative reader to predict all answers in sequence based on all selected evidence, it learns pathological dependencies among answers. to address these issues, we propose to tackle this task with a recall-then-verify framework, which separates the reasoning process of each answer with a higher level of evidence usage while also leveraging large models under the same memory constraint. some previous work argued that a reader can be confused by similar but spurious passages, resulting in wrong predictions. therefore, they proposed answer rerankers @xcite @xcite to rerank top predictions from readers. our framework is related to answer reranking but with two main differences. first, a reader typically aggregates available evidence and already does a decent job of answer prediction even without answer reranking; an answer reranker is introduced to filter out hard false positive predictions from the reader. by contrast, our answer recaller aims at finding possible answers with high recall, most of which are invalid. evidence focused on each answer is then aggregated and reasoned about by our answer verifier. it is also possible to introduce another model analogous to an answer reranker to filter out false positive predictions from our answer verifier. second, answer reranking typically compares answer candidates to determine the most valid one, while our answer verifier selects multiple valid answers mainly based on the soundness of their respective evidence but without comparisons among answer candidates. ## task formulation open-domain multi-answer qa can be formally defined as follows: given an open-ended question q, a question answering system is required to make use of evidence from a large-scale text corpus c and predict a set of valid answers {a 1 , a 2 , ..., a n }. questions and their corresponding answer sets are provided for training. evaluation to evaluate passage retrieval and reranking, we adopt the metric mrecall@k from @xcite , which measures whether the top-k passages cover at least k distinct answers (or n answers if the total number of answers n is less than k). to evaluate question answering performance, we follow @xcite to use f1 score between gold answers and predicted ones. ## rerank-then-read framework in this section, we will briefly introduce the representative and state-of-the-art rerank-then-read pipeline from @xcite for open-domain multi-answer questions, and provide empirical analysis of this framework. ## experiments 6.1 datasets we conducted experiments on two multi-answer qa datasets, whose statistics are shown in ## conclusion in this paper, we empirically analyze the problems of the rerank-then-read framework for opendomain multi-answer questions, and propose the recall-then-verify framework, which separates the reasoning process of each answer so that 1) we can have a higher level of evidence usage 2) and predicted answers are mainly based on associated evidence and are more robust to distraction by evidence of the other gold answer(s), 3) while also leveraging large models under the same memory constraint. on two multi-answer datasets, our framework significantly outperforms rerank-thenread baselines with new state-of-the-art records. pose a recall-then-verify framework that will hopefully benefit information-seeking users with an enhanced ability of comprehensive exploitation of evidence from a large-scale corpus. as our predictions are verified with textual knowledge, our system itself would not raise new significant ethical concerns. all the datasets as well as the retrieval corpus in our experiments have been widely used for research purposes, and to our knowledge, do not have any attached privacy and ethical issues. wen-tau yih, matthew richardson, christopher meek, ming-wei chang, and jina suh. 2016. the value of semantic parse labeling for knowledge base question answering. in proceedings of the 54th annual meeting of the association for computational linguistics, acl 2016, august 7-12, 2016, berlin, germany, volume 2: short papers. the association for computer linguistics. tiancheng zhao, xiaopeng lu, and kyusong lee. 2021. sparta: efficient open-domain question answering via sparse transformer matching retrieval. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, naacl-hlt 2021, online, june 6-11, 2021, pages 565-575. association for computational linguistics.