Conversational QA Dataset Generation with Answer Revision (2022)

Abstract: Conversational question-answer generation is a task that automatically generates a large-scale conversational question answering dataset based on input passages. In this paper, we introduce a novel framework that extracts question-worthy phrases from a passage and then generates corresponding questions considering previous conversations. In particular, our framework revises the extracted answers after generating questions so that answers exactly match paired questions. Experimental results show that our simple answer revision approach leads to significant improvement in the quality of synthetic data. Moreover, we prove that our framework can be effectively utilized for domain adaptation of conversational question answering.

## introduction conversational question answering (cqa) involves answering questions by considering a given text as well as previous conversations. to facilitate research on cqa, a range of datasets have been proposed in recent years @xcite @xcite @xcite . however, building a robust cqa system for a specific domain requires a large-scale domain-specific dataset; moreover, obtaining such a dataset is considerably expensive and time-consuming. to resolve this issue, in our previous study, we had proposed a conversational question-answer generation (cqag) framework that automatically creates multiturn question-answer (q-a) pairs from given passages @xcite . the framework is a two-stage architecture that adopts contextual answer extraction (cae) and conversational question generation (cqg). considering previous conversations, the cae module extracts the next question-worthy phrase from the passage, and then the cqg module generates the conversational * corresponding author question corresponding to the phrase. however, the framework has the limitation that the error may propagate to the question generation stage and even to data generation for subsequent turns if improper answers are extracted by the cae module. in this paper, we introduce a cqag framework with answer revision (cqag-ar), in which the conversational question generation with answer revision (cqg-ar) module revises the extracted answer to a more suitable one immediately after generating a question. in experiments, we synthesize cqa data using cqag-ar and then evaluate cqa models trained on these synthetic data. results reveal that answer revision by the cqg-ar module leads to absolute improvements of up to 13.4% and 15.3% in f1 score and exact match (em), respectively, for the cqa models. furthermore, finetuning the wikipedia-domain cqa model on different synthetic data increases em by up to 13.1%, showing that our framework is beneficial for cqa domain adaptation. by contrast, answer-unaware cqg synthesizes conversational questions without given answers @xcite @xcite . further, @xcite introduced a framework that first finds the location of points of interest in the passage, identifies question types, and subsequently generates conversational questions. cqag attempts to automatically construct cqa data for various domains. in our previous study (hwang and lee, 2021), we designed a vanilla cqag that generated multiturn q-a pairs based on a given passage in an autoregressive manner. however, the framework has a drawback in that the validity of the extracted answer directly affects the quality of the conversation. ## conclusion in this paper, we propose cqag-ar, which automatically synthesizes high-quality cqa data. our framework comprises cae and cqg-ar modules. considering the conversation history, the cae module extracts a question-worthy phrase from a given passage, and then the cqg-ar module generates a conversational question while revising the extracted answer to make it more suitable. experimental results show that cqag-ar outperforms baseline frameworks in terms of generating high-quality cqa data. in addition, fine-tuning a wikipedia-domain cqa system on our synthetic data for out-of-domain cases improves the model performances by significant margins.