How Many Answers Should I Give? An Empirical Study of Multi-Answer Reading Comprehension (2023)

Abstract: The multi-answer phenomenon, where a question may have multiple answers scattered in the document, can be well handled by humans but is challenging enough for machine reading comprehension (MRC) systems. Despite recent progress in multi-answer MRC, there lacks a systematic analysis of how this phenomenon arises and how to better address it. In this work, we design a taxonomy to categorize commonly-seen multi-answer MRC instances, with which we inspect three multi-answer datasets and analyze where the multi-answer challenge comes from. We further analyze how well different paradigms of current multi-answer MRC models deal with different types of multi-answer instances. We find that some paradigms capture well the key information in the questions while others better model the relation between questions and contexts. We thus explore strategies to make the best of the strengths of different paradigms. Experiments show that generation models can be a promising platform to incorporate different paradigms. Our annotations and code are released for further research.

## introduction in the typical setting of machine reading comprehension, such as squad @xcite , the system is expected to extract a single answer from the passage for a given question. however, in many scenarios, questions may have multiple answers scattered in the passages, and all the answers should be found to completely answer the questions, such as the examples illustrated in figure 1 . recently, a series of mrc benchmarks featuring multi-answer instances have been constructed, including drop @xcite , quoref (dasigi et al., 2019) and multispanqa @xcite . most current research efforts focus primarily on improving the overall qa performance on these benchmarks @xcite @xcite . yet, as far as we know, there still lacks a systematic analysis of how the phenomenon of multi-answer arises and how we can better tackle this challenge. in this paper, we systematically analyze the categorization of multi-answer mrc instances and investigate how to design a strong multi-answer mrc system. we try to answer the following research questions: (1) where does the multianswer challenge come from? (2) how do different mrc models specifically deal with the multianswer challenge? (3) how can we design better models by combining different multi-answer mrc paradigms? we first analyze existing multi-answer mrc datasets to track the origin of the multi-answer challenge. previous works have attempted to categorize multi-answer instances primarily based on the distances or relationships between multiple an-swers @xcite . yet, they did not holistically consider the interaction between questions and contexts. we observe that in some cases the number of answers is indicated in the question itself (two players in example a of figure 1 ) while in others we have no idea until we read the documents carefully (example b of figure 1 ). to better understand this challenge, we develop a taxonomy for the multi-answer phenomenon, based on how the number of answers is determined: the question itself suffices, or both the question and the passage should be taken into consideration. we annotate 6,857 instances from drop, quoref, and multispanqa based on our taxonomy and find that the procedure of dataset construction has a large influence on the expressions in the questions. most questions in crowdsourced datasets contain certain clues indicating the number of answers. by contrast, real-world information-seeking questions are less likely to specify the number of answers, which is usually dependent on the passages. we further use our annotations to examine the performance of current mrc solutions regarding the multi-answer challenge @xcite @xcite , which can be categorized into 4 paradigms, i.e., tagging, numpred, iterative and generation. we analyze their strengths and weaknesses and find that some efforts, e.g., numpred, are good at capturing the key information in the questions, while others, e.g., iter-ative, can better model the relation between questions and contexts. this motivates us to investigate better ways to benefit from different paradigms. given the complementary nature of these paradigms, we wonder whether a combination of paradigms improves performance on multi-answer mrc. we explore two strategies, early fusion and late ensemble, to benefit from different paradigms. with a generation model as the backbone, we attempt to integrate the paradigms numpred and interative, in a lightweight chain-of-thought style @xcite . experiments show that the integration remarkably improves the performance of generation models, demonstrating that genera-tion is a promising platform for paradigm fusion. our contributions are summarized as follows: (1) we design a taxonomy for multi-answer mrc instances according to how the number of answers can be determined. it considers both questions and contexts simultaneously, enlightening where the multi-answer challenge comes from. (2) we anno-tate 6,857 instances from 3 datasets with our taxonomy, which enables us to examine 4 paradigms for multi-answer mrc in terms of their strengths and weaknesses. (3) we explore various strategies to benefit from different paradigms. experiments show that generation models are promising to be backbones for paradigm fusion. ## task formulation in multi-answer mrc, given a question q and a passage p , a model should extract several spans, @xmath0, a 2 , ..., a n }(@xmath1, from p to answer q. each span, a @xmath2, corresponds to a partial answer to q, and the answer set a as a whole answers q completely. these spans can be contiguous or discontiguous in the passage. we distinguish between two terms, multi-answer and multi-span, which are often confused in previous works. multi-answer indicates that a question should be answered with the complete set of entities or utterances. multi-span is a definition from the perspective of answer annotations. in certain cases, the answer annotation of a question can be either single-span or multi-span, as explained in the next paragraph. ideally, we expect that the answers to a multi-answer question should be annotated as multi-span in the passage, where each answer is grounded to a single span, although some of them can be contiguous in the passage. for example, in q0, there are two answers, english and french, to the given question. according to the annotation guidelines of squad, one might annotate this instance with a single continuous span english and french. yet, this form of annotation is not preferred in the multi-answer mrc setting. it blurs the boundary of different answers and fails to denote explicitly the number of expected answers. thus, it is suboptimal for a comprehensive model evaluation. instead, we suggest denoting each answer with distinct spans, say, annotating this instance with two spans, english and french. with this criterion, we can encourage models to disentangle different answers. with fine-grained answer annotations, we can also assess how well a model answers a question sufficiently and precisely. this annotation criterion generally conforms to the annotation guidelines of existing multi-answer datasets, e.g., drop, quoref and multispanqa. a few instances violating the criterion are considered as bad annotations, as discussed in section 4.2. see more remarks on the task formulation in appendix a. ## taxonomy of multi-answer mrc to better understand the challenge of multi-answer, we first design a taxonomy to categorize various multi-answer mrc instances. it assesses how the number of answers relates to the question or passage provided. different from the previous works that classify questions according to the distances or relations between multiple answers @xcite , our taxonomy, taking both questions and passages into consideration, focuses on how the number of answers is determined. this enables us to analyze multi-answer questions and single-answer questions in a unified way. we illustrate our taxonomy in figure 2 and elaborate on each category as follows. question-dependent if one can infer the exact number of answers from the question without referring to the passage, this instance belongs to the question-dependent category. according to whether there are clue words that directly indicate the number of answers, this type is further divided into two sub-categories: (a) in a with-clue-words question, one can find a few words that indicate the number of answers. in q1, the word two in the question indicates that two answers are expected. q1: what are the two official languages of puerto rico? p: [...] english is an official language of the government of puerto rico. [...] as another official language, spanish is widely used in puerto rico. @xcite we group the clue words into five types: cardinal, ordinal, comparative/superlative, alternative, and other lexical semantics, as illustrated in q2: 1 light year equal to how many km? p: [...] the light-year is a unit of length used to express astronomical distances. it is about 9.5 trillion kilometres or 5.9 trillion miles. @xcite in q3, we can infer that the following question has only one answer, based on the commonsense that there is only one winner of a given super bowl. q3: who won super bowl xxxix? p: [...] the eagles advanced to super bowl xxxix, where passage-dependent in a passage-dependent instance, the question itself is not adequate to infer the number of answers. one needs to rely on the provided passage to decide how many answers are needed to answer the question. in q4, we have no idea of the number of answers solely based on the question. if we refer to the passage, we will find ten answers to the question. q4: which countries does the danube river flow through? p: [...] originating in germany, the danube flows southeast for 2,850 km, passing through or bordering austria, slovakia, hungary, croatia, serbia, romania, bulgaria, moldova and ukraine before draining into the black sea. [...] 4 analyses of multi-answer datasets we investigate existing multi-answer datasets based on our designed taxonomy to analyze where the multi-answer challenge comes from. dataset all single-ans. multi-ans. drop 3,133 2,609 524 quoref 2,418 2,198 220 multispanqa 1,306 653 653 total 6,857 5,460 1,397 ## existing multi-answer mrc models based on our categorization of the multi-answer instances, we continue to investigate how existing multi-answer mrc models perform differently on various types of multi-answer instances. we summarize current solutions into four paradigms according to how they obtain multiple answers, as illustrated in figure 3 . @xcite cast the multianswer mrc task as a sequence tagging problem, similar to named entity recognition (ner), so that the model can extract multiple non-contiguous spans from the context. @xcite first predict the number of answers k as an auxiliary task and then select the top k nonoverlapped ones from the output candidate spans. ## fusion of different paradigms from the above analysis, we can see that extractive methods can better locate exact short spans in the passage, and numpred can provide potential guidance on the number of answers. meanwhile, the generation models can better handle longer answers and are more adaptable to different forms of inputs and outputs. now an interesting question is how to combine different paradigms to get the best of both worlds. we explore two strategies for combining different paradigms: early fusion and late ensemble. the former mixes multiple paradigms in terms of model architectures while the latter ensembles the predictions of different models. we discuss our exploration of late ensemble in appendix e.1 since model ensemble is a well-explored technique. here we primarily elaborate on early fusion. we carry out a series of pilot studies to demonstrate the potential of paradigm fusion. previous works attempt to fuse two extractive paradigms, tagging and numpred @xcite . however, they only lead to marginal improvements, probably because tag-ging can already implicitly determine answer numbers well and the help of numpred is thus limited. although the performance of base-size generation models on multi-answer mrc is inferior to that of extractive ones, generation models of larger sizes show great potential with more parameters and larger pre-training corpora @xcite . more importantly, generation can easily adapt to various forms of inputs and outputs. we carry out pilot studies using a generation model as the backbone and benefiting from the ideas of other paradigms. we propose several lightweight methods to combine generation with numpred and iterative, as illustrated in figure 4 . @xcite , we guide the model with prompts indicating the number of answers. we introduce a numpred prompt sentence (nps) in the form of there are {2, 3, ...} answers/there is only one answer. we experiment with two variants, multitask and pipeline. in the multitask variant, the model outputs an nps before enumerating all the answers. in the pipeline variant, we predict the number of answers with a separate classifier and then append the nps to the question as extra guidance. ## related works compared to the vast amount of single-answer mrc datasets, the resources for multi-answer mrc are limited. aside from the datasets in section 4.1, mash-qa @xcite focuses on the healthcare domain, with 27% of the questions having multiple long answers, ranging from phrases to sentences. cmqa @xcite another multi-answer dataset in chinese, featuring answers with conditions or different granularities. for our analysis, we select two commonly-used datasets, drop and quoref, as well as a newlyreleased dataset, multispanqa. current models addressing multi-answer mrc generally fall into two paradigms: tagging (segal et al., 2020) and numpred @xcite , as explained in section 5. iterative @xcite @xcite and generation @xcite have been adopted for many types of qa tasks including knowledge base qa, multiplechoice qa, and open-domain qa. nevertheless, their performance on multi-answer mrc is less explored. in our paper, we also study how to adapt these paradigms for multi-answer mrc. apart from the exploration of model architectures for multi-answer mrc, @xcite attempt to generate multi-answer questions as data augmentation. previous works have made preliminary attempts in fusing two extractive paradigms. @xcite adopt a single-span extraction model for single-answer questions and tagging for multianswer questions; @xcite add a numpred head to the tagging framework. the predicted number of answers is used to adjust the tagging results. both strategies lead to marginal improvement over the baselines. we instead resort to genera-tion for paradigm fusion, considering its potential with larger sizes and its flexibility in inputs and outputs. ## conclusion in this paper, we conduct a systematic analysis for multi-answer mrc. we design a new taxonomy for multi-answer instances based on how the number of answers is determined. we annotate three datasets with the taxonomy and find that multi-answer is not merely a linguistic phenomenon; rather, many factors contribute to it, especially the process of data collection. with the annotation, we further investigate the performance of four paradigms for multi-answer mrc and find their strengths and weaknesses. this motivates us to explore various strategies of paradigm fusion to boost performance. we believe that our taxonomy can help determine what types of questions are desirable in the annotation process and aid in designing more practical annotation guidelines. we hope that our annota-tions can be used for more fine-grained diagnoses of mrc systems and encourage more robust mrc models.