Multi-Type Conversational Question-Answer Generation with Closed-ended and Unanswerable Questions (2022)

Abstract: Conversational question answering (CQA) facilitates an incremental and interactive understanding of a given context, but building a CQA system is difficult for many domains due to the problem of data scarcity. In this paper, we introduce a novel method to synthesize data for CQA with various question types, including open-ended, closed-ended, and unanswerable questions. We design a different generation flow for each question type and effectively combine them in a single, shared framework. Moreover, we devise a hierarchical answerability classification (hierarchical AC) module that improves quality of the synthetic data while acquiring unanswerable questions. Manual inspections show that synthetic data generated with our framework have characteristics very similar to those of human-generated conversations. Across four domains, CQA systems trained on our synthetic data indeed show good performance close to the systems trained on human-annotated data.

## introduction conversational question answering (cqa) aims to answer a question based on a given passage and previous conversation. unlike single-turn question answering (qa) @xcite @xcite , cqa encourages questioners to incrementally make follow-up questions, which is suitable for services that require active interaction between humans and systems. however, manually creating large amounts of conversations is very costly, which is a barrier to its utilization in various domains. to alleviate this issue, a few methods for conversational question generation have been studied @xcite @xcite @xcite . furthermore, we have proposed approaches for automatically synthesizing multi-turn conversational question-answer (q-a) pairs in order to build train-ing data for cqa in our previous studies @xcite . however, our previous frameworks generate only open-ended questions that cannot be answered succinctly. in real-world situations, concise answers, such as yes, no, and unknown, are essential for fast interaction and simplified conversations. in this paper, we introduce multicqag, a framework that can generate multiple types of cqa data. to enable this, we insert a generation flow for closed-ended q-a pairs to our previous framework @xcite . we also design a hierarchical answerability classification (hierarchical ac) module that collects yet another type of data -unanswerable questions -while improving data quality by removing invalid q-a pairs. in experiments, cqa systems trained on our synthetic datasets achieve an average f1 score of 77.2% for four new domains, showing a difference of only 5.4% from those trained on humanannotated data. moreover, we show by manual evaluation that our synthetic data have a data distribution similar to that of human-annotated data. the contributions of this work can be summarized as follows: in our previous study, we proposed a conversational question-answer generation (cqag) framework that automatically synthesized data for cqa given passages and that consisted of two modules: contextual answer extraction (cae) and conversational question generation (cqg) (hwang and lee, figure 1: generation pipeline of multicqag. conversation history is not used to generate the first q-a pair of a conversation (dotted line). 2021). first, the cae module extracts a potential answer span from a passage based on a previous conversation. second, the cqg module generates a conversational question for the extracted answer. during generation, the framework uses previously generated q-a pairs as the conversation history for the next generation. however, synthetic data generated by this framework only contain extractive answers that are inflexible in form. moreover, there is a risk that errors generated by the cae module can propagate to subsequent generations. to resolve this problem, we developed cqag-ar, which adopted an answer revision approach @xcite . in this framework, the cqg with answer revision (cqg-ar) module generates a question for the extracted answer span and then modifies the answer span so that it better fits the question. however, cqag-ar can only synthesize open-ended types of data and cannot generate closed-ended and unanswerable types, which are frequently used in human conversations. in this paper, we improve cqag-ar to generate those different types of data in a single framework. ## experiments we utilized coqa @xcite , a dataset for constructing cqa systems, to prove that multi-cqag generates high-quality synthetic cqa data in new domains. coqa is based on passages collected from seven different domains. among the domains, only five that constituted official training and development sets foot_0 were used in our experiments: wikipedia, for training multicqag, and four other domains for data synthesis and cqa evaluation. we generated synthetic cqa data for four domains (children's stories, literature, news, and middle and high school english exams) by using trained multicqag and four collections of passages extracted from coqa training and development sets. see appendix a for examples of synthetic data and appendix b for implementation details. the synthetic data were evaluated through two methods: extrinsic evaluation using the cqa task, and human evaluation. during the extrinsic evaluation, we trained t5-based cqa systems for four different domains on our synthetic data, and we evaluated each system using the coqa test set 2 for corresponding domains. ## conclusion in this paper, we introduce a multi-type data synthesis framework for cqa with individual generation flows for open-ended, closed-ended, and unanswerable questions. our framework has a unique twolevel classification module to filter invalid q-a pairs and produce unanswerable questions simultaneously. by cqa system training and manual evaluations, we proved that the data synthesized with our framework have a quality comparable to that of human-generated cqa data.