A mbig QA : Answering Ambiguous Open-domain Questions (2020)

Abstract: Ambiguity is inherent to open-domain question answering; especially when exploring new topics, it can be difficult to ask questions that have a single, unambiguous answer. In this paper, we introduce AmbigQA, a new open-domain question answering task which involves finding every plausible answer, and then rewriting the question for each one to resolve the ambiguity. To study this task, we construct AmbigNQ, a dataset covering 14,042 questions from NQ-open, an existing open-domain QA benchmark. We find that over half of the questions in NQ-open are ambiguous, with diverse sources of ambiguity such as event and entity references. We also present strong baseline models for AmbigQA which we show benefit from weakly supervised learning that incorporates NQ-open, strongly suggesting our new task and data will support significant future research effort. Our data and baselines are available at https://nlp.cs.washington.edu/ambigqa .

## introduction in the open-domain setting, it can be difficult to formulate clear and unambiguous questions. for example, figure 1 shows a google search query @xcite that, perhaps surprisingly, has two possible interpretations given the evidence in wikipedia. although open-domain question answering (qa) systems aim to answer any factoid question @xcite , existing methods assume questions have a single welldefined answer. nonetheless, ambiguity arises frequently in open-domain qa, where questions are written during information gathering (e.g., search queries) without knowledge of the answer. as we will see in section 4, over 50% of the questions we sampled from a set of google search queries are ambiguous. furthermore, identifying ambiguities is difficult both for humans and machines. as shown in figure 1 , ambiguity is a function of both the question and the evidence provided by a large text corpus. to study this challenge, we introduce am-bigqa (answering ambiguous open-domain questions), a new task which involves disambiguating and answering potentially ambiguous questions. specifically, the model must (1) find a set of distinct, equally plausible answers to the question, and (2) provide minimal yet unambiguous rewrites of the question that clarify the interpretation which leads to each answer. figure 1 shows two such disambiguated questions and their answers. to support the study of this task, we construct a dataset called ambignq using 14,042 questions from an open-domain version of natural ques-tions @xcite , denoted nq-open. for each question, annotators search for, timedependency (13%) when does the new family guy season come out? q: when does family guy season 16 come out? / a: october 1, 2017 q: when does family guy season 15 come out? / a: september 25, 2016 q: when does family guy season 14 come out? / a: september 27, 2015 multiple sub-questions (3%) who was british pm and viceroy during quit india movement? q: who was british viceroy during quit india movement? / a: victor hope q: who was british pm during quit india movement? / a: winston churchill to establish initial performance levels on this data, we present a set of strong baseline methods. we extend a state-of-the-art qa model @xcite with three new components: (1) set-based question answering with a sequence-tosequence model, (2) a question disambiguation model, and (3) a modification to democratic cotraining @xcite which leverages the partial supervision available in the full nq-open dataset. we also do an ablation study and qualitative analysis, which suggest there is significant room for future work on this task. to summarize, our contributions are threefold. 1. we introduce ambigqa, a new task which requires identifying all plausible answers to an open-domain question, along with disambiguated questions to differentiate them. our work is built upon an open-domain version of natural questions @xcite , denoted nq-open, composed of questions posed by real users of google search, each with an answer drawn from wikipedia. nq-open has promoted several recent advances in open-domain question answering @xcite @xcite @xcite . nonetheless, @xcite report that the answers to such questions are often debatable, and the average agreement rate on nq-open test data is 49.2%, foot_0 in large part due to ambiguous questions. in this work, we embrace this ambiguity as inherent to information seeking open-domain qa, and present the first methods for returning sets of answers paired with different interpretations of the question. clarification questions have been used to study question ambiguity in other settings. research on community q&a @xcite @xcite studies finding underspecification in the question, but it does not find the answer to the original question. in recent work, @xcite study clarification of questions that are intentionally annotated with pre-specified entity reference ambiguities. @xcite and @xcite use clarification questions to refine intents of simple query logs without immediately apparent information needs (e.g., single keywords like dinosaur foot_1 ). in contrast, we study open-domain factoid questions asked by real users: these present clear information needs, but carry diverse naturally occurring ambiguities (see question rewriting is a novel, well-defined task which we propose for differentiating distinct answers. to the best of our knowledge, it has not been studied for resolving ambiguity; we are only aware of @xcite which use question rewriting to convert conversational questions into self-contained questions. ## model to set initial performance levels on ambignq, we present a baseline ambigqa model combining ideas from recent advances in open-domain qa @xcite and generation @xcite . given a prompt question q, our model predicts answers y 1 ..y n , and generates corresponding questions x 1 ..x n conditioning on q, the answers y 1 ..y n , and the evidence passages. a novel cotraining step also allows the model to leverage the partial supervision available in nq-open. ## experiments we describe the baseline models used in our experiments, followed by results and ablations. implementation details and hyperparameters of all models are provided in appendix d. ## conclusion & future work we introduced ambigqa, a new task that involves providing multiple possible answers to a potentially ambiguous open-domain question, and providing a disambiguated question corresponding to each answer. we constructed ambignq, a dataset with 14,042 annotations on nq-open questions. our analysis shows the dataset contains diverse types of ambiguity, often not visible from the prompt question alone. we also introduced a first base-line model for producing multiple answers to opendomain questions, with experiments showing its effectiveness in learning from our data while highlighting possible areas for improvement. future research developing on ambigqa models may include explicitly modeling ambiguity over events and entities or in the retrieval step, as well as improving performance on the difficult problems of answer recall and question disambiguation. furthermore, future work may build on the ambigqa task with more open-ended approaches such as (1) applying the approach to qa over structured data (such as ambiguous questions that require returning tables), (2) handling questions with no answer or ill-formed questions that require inferring and satisfying more complex ambiguous information needs, and (3) more carefully evaluating usefulness to end users.