C onv RGX : Recognition, Generation, and Extraction for Self-trained Conversational Question Answering (2023)

Abstract: Collecting and constructing human-annotated corpora for training conversational question-answering (CQA) models has recently been shown to be inefficient and costly. To solve this problem, previous works have proposed training QA models with automatically generated QA data. In this work, we extend earlier studies on QA synthesis, and propose an efficient QA data generation algorithm under conversational settings. Our model recognizes potential dialogue topics, generates corresponding questions, and extracts answers from grounding passages. To improve the quality of generated QAs and downstream self-training of CQA models, we propose dropout and agreement-based QA selection methods. We conduct experiments on both data augmentation and domain adaptation settings. Experiments on the QuAC and Doc2Dial tasks show that the proposed method can significantly improve the quality of generated QA data, and also improves the accuracy of self-trained CQA models based on the constructed training corpora.

## introduction recent progress on pre-trained language models @xcite @xcite has significantly improved the performance of different natural language understanding tasks, including question answering (qa). however, task-specific fine-tuning of pretrained models still requires human-annotated training corpora, especially for qa. for example, training a qa model on the wikipedia domain needs a training set of over 80,000 human-annotated question-answer pairs @xcite ). annotating such a training corpus is too costly to be generalized for other domains and qa tasks. moreover, many real-life agents answer questions in a † these authors contributed equally to this work and share first authorship. due to the limited amount of labeled training data and questions for conversational qa tasks being more complicated, there is a significant performance gap between single-turn and conversational qa models. as a coarse reference rather than a direct comparison, single-turn qa models achieve over 90% exact match score on squad @xcite , and the accuracy of most cqa models is below 70% on the doc2dial benchmark @xcite . to address this problem of insufficient conversational qa for training, we propose an automatic conversational question-answering data annotation method. inspired by the recognition-generationextraction (rgx) pipeline @xcite , we design a conversation generation algorithm (named convrgx), which generates dialogues based on grounding documents. to generate a question and the corresponding answer in a conversation, the model first recognizes a possible dialogue topic from the grounding document, which provides information about the answer. given the topic, a number of questions are generated. we then use a pre-trained question-answering model to verify the generated questions by comparing the selected dialogue topic and the answer extracted by the qa model given the generated questions, or the agreement between cqa models with different dropout. among all generated qa pairs, we filter out lowquality data, samples high-quality qa pairs as the current dialogue turn, and continue to generate the next question. compared to the baseline rgx model, we improve the answer recognition module and apply a dropout-based data selection strategy to improve the model under conversational settings. to prove the effectiveness of convrgx, we evaluate the generated qa data along different dimensions. we first evaluate the question quality using bleu @xcite , rougel @xcite , and q-metric @xcite . experiments show that convrgx generates high-quality questions. we also conduct self-training for the cqa models with the generated qa data. experiments show that the data generation and selection framework can constantly improve the data synthesis quality and qa self-training performance. self-trained question answering recent work have studied the potential for improving qa models with question generation. a question generator benefits mutual information-based qa @xcite , in-domain data augmentation @xcite @xcite , and outof-domain adaptation of qa models. @xcite and @xcite introduced qa generation frameworks for self-trained question answer-ing. @xcite proposed an end-to-end qa generation model, and @xcite showed that the quality and diversity of generated qa can be improved by difficult qa cases. @xcite proposed a cooperative self-training strategy that benefits both question generation and answering. @xcite ; @xcite presented additional applications for qa generation systems. ## method for automatically generating conversational qa data on unlabeled grounding documents, we propose a 3-step pipeline named convrgx. in order to generate a dialogue turn, we first recognize the upcoming possible topic from the grounding document and then generate a number of candidate questions. the generated questions are then filtered, and a pre-trained cqa model is applied to predict refined answers for the generated questions. the pipeline is illustrated in figure 1 . in this section, we introduce the details of each step of convrgx. ## analysis in this section, we report results with extractivebased cqa models trained with only generated dialogues on the doc2dial dataset to further explore the contribution of different factors to the 7 the em values are close to zero but not exactly zero because there are some sentences overlapping among the documents in the training dataset and testing dataset even under the unseen documents setting. quality of synthesized conversations. ## conclusion we propose convrgx, an automatic cqa data annotation method extended from the recognitiongeneration-extraction (rgx) framework for conversational applications, which can generate highquality cqa data that can be used for question generation and data augmentation. we demonstrate the effectiveness of convrgx on standard conversational benchmarks, which show improvements over current data generation and augmentation methods for both question quality and self-training performance. in summary, convrgx presents a scalable and effective way to approach cqa problems that have limited human annotation.