ASQA : Factoid Questions Meet Long-Form Answers (2022)

Abstract: Recent progress on open domain factoid question answering (QA) does not easily transfer to the task of long-form QA, where the goal is to answer questions that require in-depth explanations. The hurdles include a lack of high-quality data and the absence of a well-defined notion of an answer’s quality. In this work, we address these problems by releasing a novel dataset and a task that we call ASQA (Answer Summaries for Questions which are Ambiguous); and proposing a reliable metric for measuring performance on ASQA. Our task focuses on ambiguous factoid questions which have different correct answers depending on the interpretation. Answers to ambiguous questions should combine factual information from multiple sources into a coherent long-form summary that resolves the ambiguity. In contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear notion of correctness: a user faced with a good summary should be able to answer different interpretations of the original ambiguous question. Our analysis demonstrates an agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines.

## introduction in the last few years, the factoid question answering (qa) task-extracting short answers to factoid questions-has witnessed significant progress @xcite @xcite @xcite . the progress was achieved in large part thanks to (i) the availability of high-quality datasets @xcite @xcite @xcite , and (ii) a well-defined notion of correctness. a key challenge for ongoing research now lies in longform question answering where the goal is to generate detailed explanations in response to questions that require elaborate and in-depth answers. there is much less data available for the task of long-form qa. one of the primary data sources is the eli5 dataset @xcite ) that pairs open-ended questions with paragraph-long answers written by users of the "explain like i'm five" reddit forum. however, questions in eli5 are very general (e.g., "how can different animals perceive different colors?") and can be answered in myriad different ways, making it hard to define objective criteria for a good answer. as a result, @xcite identify several hurdles in using this data towards meaningful modeling progress, including a lack of reliable evaluation metrics. in this work, we address the lack of data sources and unreliability of evaluations by constructing a long-form qa dataset for factoid questions. our paper is motivated by the work of @xcite who observe that more than half of the factoid questions that occur naturally are ambiguous. for example, a seemingly simple question: "who was the ruler of france in 1830?" is ambiguous because there were two rulers of france in 1830. @xcite collected the ambigqa dataset that connects ambiguous factoid questions with disambiguations: pairs of disambiguated questions and unique short answers to these questions (see example on the right side of figure 1 ). we note, however, that ambiguous questions often arise when a user lacks background knowledge about why there might be multiple answers to their question, and how those answers relate to each other. thus, the list of disambiguations may not be satisfactory for the user. for example, the fact that in 1830 the ruler of france changed due to the revolution is highly salient but is not captured in figure 1 : the input questions in asqa are sourced from ambigqa. long-form answers must be sufficient to answer disambiguated questions from ambigqa (short answers are marked in blue and green), and should introduce additional knowledge from wikipedia (highlighted in red) to resolve ambiguity and clarify the relationship between different short answers. the dr score we propose combines rouge and disambiguation-accuracy (disambig-acc) metrics, overcoming the issues with long-form qa evaluation outlined by @xcite . in this paper, we argue the importance of generating long-form answers to ambiguous factoid questions. in that, we present asqa (answer summaries for questions which are ambiguous)a novel dataset that pairs each ambiguous question from ambigqa with a crowdsourced long-form answer. 1 the answers we collect aim to (i) explain the source of ambiguity in the question, and (ii) connect all the valid short answers into a coherent passage. an example asqa instance is shown in figure 1 . the main feature of asqa is a combination of (i) a well-defined notion of correctness pertinent to factoid qa and (ii) the complexity of long-form qa. first, observe that a good answer to an ambiguous question should be sufficient for the user to answer different interpretations of the question. this observation induces a notion of correctness that is conceptually similar to the conventional accuracy in factoid qa. second, to answer an ambiguous question, a system needs to retrieve a diverse set of documents that talk about different interpretations of the question and synthesize this information into a coherent summary. thus, the key challenges of long-form qa-precise retrieval and high-quality summarization-are present in asqa. contributions overall, our work makes several contributions: ## related work in this section, we describe relevant works that propose new tasks, datasets, and methods for qa and summarization problems. extractive qa much of the existing work on question answering, including reading comprehension @xcite @xcite , open-domain qa @xcite and dialog-based qa @xcite , assumes that questions have unique answers. @xcite relax this assumption and propose a task that aims at identifying all possible short answers to the ambiguous subset of the open-domain version of the nq dataset, denoted nq-open @xcite . the ambigqa dataset constructed by @xcite serves as a building block of the present work and we provide more details on this dataset in section 3. another related effort is the conditionalqa task @xcite that requires systems to identify conditions under which the extracted answers are valid. unlike the asqa task, the answers in condi-tionalqa come from a document provided in advance and do not need to be summarized into a single response. generative qa extractive models achieve good results when the answer to the question is readily available on the web. however, in many settings, including ambiguous factoid questions, a system needs to combine information from many (unknown) sources to present the answer to the user in a convenient way. hence, in this work, we focus on the generative qa setting where a model needs to generate a textual answer rather than extract it. datasets for generative qa include narra-tiveqa @xcite and coqa @xcite , but the average answer length in these datasets is small: 4.7 and 2.7 tokens, respectively. the ms marco natural language generation (ms-nlg) dataset by @xcite combines both extractive and generative tasks and contains slightly longer human-generated answers (usually, a sentence-long) that can be read by a smart assistant. @xcite proposed a more challenging task of answering open-ended (e.g., "why?") questions. they scraped the "explain like i'm five" reddit forum and released a dataset of ∼272k questions, where each question is supplied with several paragraph-long answers generated by the reddit users. we overview the differences between asqa, eli5 and ms-nlg in section 3.3. recently, large language models such as gpt-3 @xcite have been successfully applied to the task of long-form qa using the eli5 dataset @xcite . for this, a two-step human-in-the-loop approach was involved: first, demonstrations of annotators navigating the web to write answers were collected; second, a reward model @xcite was trained by manual pairwise comparisons of answers. in asqa, relevant passages for the answer are already provided by the annotators and we show that the pro-posed dr score correlates well with the human judgment of answer quality. using this automated metric in place of the reward model in the approach of @xcite is a potential direction for future work. summarization given a set of documents relevant to the question (either ground truth or obtained using retrieval) the problem of generating a long-form answer reduces to query-based multi-document summarization. a small-scale dataset for this task was introduced as part of the duc tasks @xcite . recent work on building large-scale datasets has instead focused either on query-based summarization from a single document @xcite or on multi-document summarization without queries @xcite . in addition to the qa task, the asqa dataset is suitable for the evaluation of systems' accuracy in the summarization setting, where the ground-truth passages containing the relevant information are assumed to be given. ## asqa task and data in this section, we introduce the asqa task and the underlying data-collection process. the asqa task is illustrated in figure 1 . the goal of the task is to write a comprehensive paragraph-long answer â to a given ambiguous question q. source data we build asqa on top of the subset of ambiguous questions identified in the am-bigqa dataset. out of a total of 14,042 am-bigqa questions, 7,207 are identified as ambiguous by at least one ambigqa annotator. each of these ambiguous questions q is paired with a list of n disambiguations {(x i , y i )} n @xmath0, where x i denotes a disambiguated question and y i denotes q1: who was the ruler of france until 2 august 1830? a1: charles x q2: who was the ruler of france after 9 august 1830? a2: louis-philippe i louis-philippe was sworn in as king louis-philippe i on 9 august 1830. upon his accession to the throne, louis philippe assumed the title of "king of the french" […] wikipedia titles: louis philippe i list of heads of state of france list of french monarchs link to the wikipedia page use this box to copy and paste supporting sentence(s) from wikipedia. use a separate box for each piece of knowledge add a new piece of knowledge please enter the long answer here who was the ruler of france in 1830? ambiguous question long answer wikipedia pages with more info box for additional knowledge task: write a coherent and detailed answer to the question below input output optional feedback. if you see anything wrong with this task, let us know here feedback field disambiguations louis philippe i was king of the french from 1830 to 1848. […] he was proclaimed king in 1830 after his cousin charles x was forced to abdicate by the july revolution […] a unique short answer to x i . the number of disambiguations ranges from 2 to 46 per ambiguous question. to ensure that it is feasible to put all this information into a coherent story, we remove 417 questions with more than six disambiguations from consideration, thereby focusing on 6,790 am-bigqa instances that we use as a starting point for building our task. ## asqa metrics in this section, we introduce metrics that we propose to evaluate performance on the asqa task. ## experimental setup we now describe the baseline models and human answers used in our experiments. ## results we evaluate all models introduced above in the automated evaluations. additionally, we conduct a small-scale human study involving a subset of models to provide some verification of the automated evaluation results. specifically, our human study involves four model outputs (jpr@1, t5-c, t5-o-1, t5-o-5) and two sets of human-generated answers (hp-w/o-c, hp-w/-c) that are juxtaposed on a subset of 45 randomly chosen questions from the development set of asqa. for each of the questions, six target answers are split into three pairs and pairwise comparisons are conducted by authors of this paper in a blind manner. importance of retrieval models that take the output of a retrieval system (t5-o-1/3/5) perform much stronger than the closed-book model (t5-c) on both automated metrics and the human evaluation. t5-o-1 outperforms t5-c by 20.0 points on human evaluation (ho) and by 12.8 points on dr. t5-o-5 outperforms t5-c by 15.6 points on ho and by 17.0 points on dr. following @xcite , we also experimented with a random retrieval baseline where, during inference, the model was provided randomly selected retrieved passages from the training set. this baseline gets a dr of only 7.8, further confirming that, different from eli5, retrieval is very important for asqa. ## analysis we now conduct additional analysis that provides insights on the asqa task. ## conclusion in contrast to existing datasets for long-form qa, asqa admits a clear notion of correctness that we use to define an overall metric of performance (dr). our empirical evaluations demonstrate that dr correlates well with the human judgment; and there is a large gap between human performance and the strong baselines. thus, we believe that asqa is an appealing task for the qa community. our analysis suggests that strong performance on asqa is contingent upon both high-quality retrieval and summarization. these aspects constitute important directions for future work on asqa.