Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction (2021)

Abstract: In open-domain question answering, questions are highly likely to be ambiguous because users may not know the scope of relevant topics when formulating them. Therefore, a system needs to find possible interpretations of the question, and predict one or multiple plausible answers. When multiple plausible answers are found, the system should rewrite the question for each answer to resolve the ambiguity. In this paper, we present a model that aggregates and combines evidence from multiple passages to adaptively predict a single answer or a set of question-answer pairs for ambiguous questions. In addition, we propose a novel round-trip prediction approach to iteratively generate additional interpretations that our model fails to find in the first pass, and then verify and filter out the incorrect question-answer pairs to arrive at the final disambiguated output. Our model, named Refuel, achieves a new state-of-the-art performance on the AmbigQA dataset, and shows competitive performance on NQ-Open and TriviaQA. The proposed round-trip prediction is a model-agnostic general approach for answering ambiguous open-domain questions, which improves our Refuel as well as several baseline models. We release source code for our models and experiments at https://github.com/amzn/refuel-open-domain-qa .

## introduction open-domain question answering (qa) is the task of answering questions using a collection of passages with diverse topics @xcite @xcite . open-domain questions are highly likely to be ambiguous because people may not have the knowledge of relevant topics when formulating them. for example, in figure 1 , the prompt question "what's the most prompt question (google search query): what's the most points scored in an nba game? disambiguated qa pairs: q1: what's the most points scored in an nba game by combined team? / a1: 370 q2: what's the most points scored in an nba game by a single team? / a2: 186 q3: what's the most points scored in an nba game by an individual? / a3: 100 relevant wikipedia page 1: the highest-scoring regular season game is the triple-overtime game between ... the two teams combined to score 370 points, with the pistons defeating the nuggets 186-184 ... relevant wikipedia page 2: wilt chamberlain scored an nba-record 100 points ... figure 1 : an example from the ambigqa @xcite dataset. the prompt question is gathered from google search queries and has three interpretations upon reading wikipedia. disambiguated qa pairs are the full set of acceptable answers, paired with the disambiguated rewriting of the prompt question. there are three main drawbacks to spanse-qgen. firstly, a complete coverage of all relevant passages is essential for predicting all plausible answers of the ambiguous question. however, spanseqgen only takes 8 passages for answer prediction so some of the most informative passages might be excluded. secondly, for the question disambiguation subtask, there is a mismatch between question generation pre-training on nq-open and question disambiguation finetuning on ambigqa -there is no question to disambiguate in question generation pre-training, which makes the pre-training task somewhat misaligned with fine-tuning. thirdly, spanseqgen predicts a much smaller average number of answers compared to the ground truth data (1.17 vs. 2.19). to address these issues, we propose refuel, round-trip evidence fusion via generation with retrieval, a new framework for answering ambiguous open-domain questions. to ensure a broad coverage of relevant knowledge of the question, refuel reads 12 times more passages (100 in our experiments) than spanseqgen by using fusionin-decoder (izacard and grave, 2020) that processes each passage individually in the encoder, and then fused their encodings together in the decoder. for the question disambiguation subtask, we propose a token-deletion pre-training task to transform nq-open into an "ambiguous" qa setting by randomly deleting an informative span for each question. thus, pre-training and fine-tuning tasks are well aligned. additionally, we add an insertionbased weighted loss to emphasize the newly inserted tokens in the disambiguated question, which helps the model on learning to resolve the ambiguity. finally, we propose a round-trip prediction approach to find additional interpretations that re-fuel fails to predict in the first pass. we contin-uously feed the generated questions into refuel until there are no new answers predicted from our model. while this round-trip prediction can improve the recall of answers, we refine the quality of predicted qa pairs by filtering them with the conditional probability of the answers estimated by an answer-generation model. our refuel achieves a new state-of-the-art on the ambigqa dataset, outperforming the previous best model spanseqgen by 9.1% in answer prediction f1 and 4.4% in edit-f1 score for question disambiguation. when directly doing inference on nq-open and triviaqa, refuel not only predicts the single answer precisely but also finds multiple interpretations if the question is ambiguous. moreover, human evaluation shows that refuel can correctly generate more qa pairs on all three datasets. finally, the proposed round-trip prediction is a model-agnostic general approach for answering ambiguous questions, which improves our refuel as well as several baseline models up to 3.7% for the overall performance. the main contributions of this work, which are fundamental to significantly push the state-of-theart in answering ambiguous questions, can be summarized as follows: the insertion-based weighted loss further helps to capture answer-relevant constraints. 3. we propose a round-trip prediction approach to find more interpretations missed in the first prediction pass, which we further refine using a conditional-probability-based filtering approach. ## refuel refuel answers questions through a three-step process illustrated in figure 2 : figure 2 : overall pipeline of refuel. refuel firstly retrieves question-relevant passages (section 2.1). then it generates first-pass qa pairs through the answer prediction (ap) module and question disambiguation (qp) module (section 3). finally, generated disambiguated questions q d are further taken as the input of our pipeline to find more interpretations (round-trip prediction). if the generated question q d still has multiple interpretations, the newly predicted answers will receive their own questions (section 2.3). ## related work open-domain question answering is answering factoid questions using a huge collection of documents such as wikipedia pages @xcite @xcite @xcite . we are motivated by the recent proposed question ambiguity problem in open-domain qa @xcite . different from the existing formulation of open-domain qa that each question only has a single answer, the proposed ambigqa task requires to predict a single answer or a set of disambiguated qa pairs depending on the ambiguity of the input question. they also propose the first model spanse-qgen to this task, which firstly uses the dense passage retriever @xcite to retrieve question-relevant passages, and then adopts a retrieval-augmented generation method @xcite to disambiguated qa pairs. our refuel follow @xcite 's task formulation and overall pipeline, but there are three differences between our refuel and spanseq-gen: (1) refuel takes the architecture of fusionin-decoder @xcite ) that can effectively use a large number of passages to uncover more candidate interpretations of the ambiguous question. (2) we propose a token-deletion pretraining task to reduce the mismatch between pretraining and fine-tuning for question disambiguation. the insertion-based weighted loss further helps to capture answer-relevant constraints. ( 3 ) we propose a model-agnostic round-trip prediction approach to find more interpretations missed in the first prediction pass, which we further refine using a conditional-probability-based filtering approach. ## conclusion in this paper, we present refuel to answer ambiguous open-domain questions. refuel is a generative approach to aggregate and combine evidence from multiple passages for multiple rounds which can find more and better interpretations. refuel achieves a new state-of-the-art on am-bigqa, and shows competitive performance on nq-open and triviaqa. the proposed round-trip prediction is a general approach for answering ambiguous open-domain questions, which improves our refuel as well as several baseline models.