CMQA : A Dataset of Conditional Question Answering with Multiple-Span Answers (2022)

Abstract: Forcing the answer of the Question Answering (QA) task to be a single text span might be restrictive since the answer can be multiple spans in the context. Moreover, we found that multi-span answers often appear with two characteristics when building the QA system for a real-world application. First, multi-span answers might be caused by users lacking domain knowledge and asking ambiguous questions, which makes the question need to be answered with conditions. Second, there might be hierarchical relations among multiple answer spans. Some recent span-extraction QA datasets include multi-span samples, but they only contain unconditional and parallel answers, which cannot be used to tackle this problem. To bridge the gap, we propose a new task: conditional question answering with hierarchical multi-span answers, where both the hierarchical relations and the conditions need to be extracted. Correspondingly, we introduce CMQA, a Conditional Multiple-span Chinese Question Answering dataset to study the new proposed task. The final release of CMQA consists of 7,861 QA pairs and 113,089 labels, where all samples contain multi-span answers, 50.4% of samples are conditional, and 56.6% of samples are hierarchical. CMQA can serve as a benchmark to study the new proposed task and help study building QA systems for real-world applications. The low performance of models drawn from related literature shows that the new proposed task is challenging for the community to solve.

## introduction question answering (qa) is a challenging benchmark task, which can drive the development of natural language understanding (nlu) methods and has significant utility to users @xcite * means both authors contributed equally to this research. @xcite ). this research area has made significant progress with many sizable datasets and standard benchmarks. notably, the span-extraction task is one of the most studied subtasks because of its wide applicability and easy evaluation characteristics. most existing span-extraction qa datasets @xcite @xcite @xcite only contain single-span samples, where the answer is a single text span in the context. however, limiting the answer to be a single span in the context might be restrictive since the answer of some questions can be multiple text spans in real-world qa scenarios. take the case in figure 1 for example, given the question asking about 'drugs for face acne redness', the corresponding drugs (green and blue highlighted text spans) appear in different parts of the context. a single span including all these drugs is extremely long, thus it is more suitable to use multi-span answers to answer this question. moreover, when building the qa system for a real-world application, we found that the multi-span answers often appear with two characteristics: first, multi-span answers can be caused by users lacking related domain knowledge and asking ambiguous questions. in such cases, questions might need multiple conditions to specify the circumstance. for example, in figure 1 , the user asked a brief question about 'drugs for acne redness'. since different drugs are needed according to the severity of symptoms:('a single appearance' and 'the acne is much and continuous into pieces'), the question is answered separately. it is misleading to give all answer spans without distinguishing the conditions. moreover, answers of different conditions might be contradictory in some samples. for example, a user asked about the battery capacity of the iphone. since the battery capacity of the iphone is different according to models, the multi-span an-gs for face age of acne. ms are not containing inhibiting the content f inflammasphate gel 4 ment. both yer of film e it for 3-4 s, you need y hand. esiangle area, ding bactebophlebitis. reatment if large area s more seriecessary to r hands freed. ## related work most existing span-extraction qa datasets only contain single-span samples, which might be restrictive in some real-world qa scenarios. an important reason for this gap is the question collection methods when building the datasets. question collection methods of qa datasets can generally be classified into two categories: creating questions by annotators and collecting questions from real-world users. most span-extraction qa datasets consist of questions created by annotators. some span-extraction qa datasets' questions are written by annotators who have first read the context containing the answer. for example, squad [16] tasks crowdworkers with asking up to 5 questions about each paragraph and highlighting the corresponding answers in the paragraph. hotpotqa [23] tasks crowdworkers with asking a question about two given paragraphs from different wikipedia pages and providing the answers. tasking annotators with questioning and answering at the same time is suboptimal, which might cause questioners to ask questions based on a text span in the context. the asked questions are often simple reformulations of sentences in the context. to avoid this problem and create more natural and challenging questions, newsqa [19] only provides news article's headlines and its summary points to questioners. the full context is unseeable. similarly, quac [4] also prevents the questioner from seeing the full context. though effort has been made, there are inevitably differences between the human-created questions and the questions asked by users in real-world scenarios. the concerns are as follows: first, the user's question is often not based on a certain context in real-world qa scenarios. second, crowdworkers might already know the form of the answer (single text span), which hints that they should ask questions with a single text span answer. last, the problems created are often high quality, while a real-world question might sometimes be ambiguous. a general term for a class of things, such as foods containing certain nutrients and people with certain characteristics. the hierarchical relations among answers of different granularities are also provided. as shown in figure 1 , labeled text spans in cmqa consist of three types: condition, coarse and fine (highlighted in the context). labeled relations consist of two types: condition-answer and coarse-fine. a structured answer can be easily derived from these labels, which is clear and accurate. furthermore, such labels are very helpful to reduce the burden of users if they want to read the full context. the final release of cmqa consists of 7,861 multi-span qa pairs and 113,089 labels, where 50.4% of samples are conditional, and 56.6% are multi-granularity. in addition, we establish models as the baseline of cmqa. traditional single-span qa models which search for the most likely token as the start/end of the answer are unsuitable for extracting multi-span answers, thus, the span extraction is cast as a sequence tagging problem @xcite . and we use competitive relation extraction methods drawn from related literature @xcite for predicting relations. experimental results show that it is very difficult to extract all spans correctly in a sample. the error analysis shows that the main challenge is to judge whether a span of the correct entity type is an answer. moreover, swers will contradict each other without specifying the condition. second, multiple answers in a sample might belong to different granularities, and there are often hierarchical relations among them. for example, in figure 1 , the answer span 'topical anti-acne drugs containing antibiotics' is a type of drug, while 'clindamycin phosphate gel' and 'mupirocin ointment' are two specific drug names of this type. if we only give all these answer spans in parallel, the user cannot get the granularity and hierarchical information of these drug names. in this case, we need to provide the answer granularity and the hierarchical relations for accuracy. the analysis results in in this paper, we introduce conditional multiple-span chinese question answering dataset (cmqa) to track the new proposed task. specifically, we pick out samples need to be answered with multiple text spans and use a new label strategy to annotate them. we labeled both answers and conditions if the sample is conditional. moreover, the answer spans are labeled with different granularity: coarse and fine. fine means the answer is a specific thing, such as a specific time, person, to name a few. coarse means the answer span is a general term for a class of things, such as foods containing certain nutrients and people with certain characteristics. the hierarchical relations among answers of different granularities are also provided. as shown in figure 1 , labeled text spans in cmqa consist of three types: condition, coarse and fine (highlighted in the context). labeled relations consist of two types: condition-answer and coarse-fine. a structured answer can be easily derived from these labels, which is clear and accurate. furthermore, such labels are very helpful to reduce the burden of users if they want to read the full context. the final release of cmqa consists of 7,861 multi-span qa pairs and 113,089 labels, where 50.4% of samples are conditional, and 56.6% are multi-granularity. in addition, we establish models as the baseline of cmqa. traditional single-span qa models which search for the most likely token as the start/end of the answer are unsuitable for extracting multi-span answers, thus, the span extraction is cast as a sequence tagging problem @xcite . and we use competitive relation extraction methods drawn from related literature @xcite ) for predicting relations. experimental results show that it is very difficult to extract all spans correctly in a sample. the error analysis shows that the main challenge is to judge whether a span of the correct entity type is an answer. moreover, though the relation model can achieve high performance on some traditional relation extraction datasets, the performance on our task is extremely poor, which makes giving structured answers like the case in figure 1 difficult. the poor model performance demonstrates that the new proposed task is challenging for the community to solve. our contributions can be summarized as follows: besides manually created questions, some qa datasets use questions collected from queries from search engines. the answer form of these datasets is usually free text instead of text spans. for example, ms marco (nguyen et al., 2016) contains queries sampled from the bing search engine. dureader @xcite , which contains queries from baidu search logs, chooses the freetext answer form, both use the free-text answer form and bleu @xcite score as the evaluation metric. though there are datasets with questions from search engines that use text span as the answer form, the span length is usually longer than phrase level. moreover, a screening mechanism is often used to filter questions. for example, wikiqa @xcite uses a single sentence as the answer form, and questions that cannot be answered with a single sentence are abandoned during the construction. natural questions @xcite filter out questions cannot be answered with an entity or explanation. moreover, natural questions provide long answers (paragraphs, tables, list items, or whole lists) besides short answers. a question might have a long answer but no short answer. these datasets show that a single span, especially a phrase-level text span, is not enough to answer real-world questions. ## dataset construction in this section, we describe the data construction process of cmqa. ## data analysis in this section, we analyze the label and question properties of the new dataset. ## experiments this section establishes models using methods drawn from related literature as the baseline of cmqa and analyzes the experiment results. ## conclusion in this paper, we propose a new challenge: conditional question answering with hierarchical multispan answers, which might be widespread in multispan qa in real-world scenarios. moreover, we introduce cmqa, which contains conditional and hierarchical samples to study the new proposed task. data analysis and experimental results show the main characteristics and challenges of cmqa, and the poor model performance demonstrates that the proposed task is challenging for the community to solve. we believe cmqa can serve as a benchmark to study the new proposed task and help build more reliable and sophisticated qa systems.