Dialogizer: Context-aware Conversational- QA Dataset Generation from Textual Sources (2023)

Abstract: To address the data scarcity issue in Conversational question answering (ConvQA), a dialog inpainting method, which utilizes documents to generate ConvQA datasets, has been proposed. However, the original dialog inpainting model is trained solely on the dialog reconstruction task, resulting in the generation of questions with low contextual relevance due to insufficient learning of question-answer alignment. To overcome this limitation, we propose a novel framework called Dialogizer, which has the capability to automatically generate ConvQA datasets with high contextual relevance from textual sources. The framework incorporates two training tasks: question-answer matching (QAM) and topic-aware dialog generation (TDG). Moreover, re-ranking is conducted during the inference phase based on the contextual relevance of the generated questions. Using our framework, we produce four ConvQA datasets by utilizing documents from multiple domains as the primary source. Through automatic evaluation using diverse metrics, as well as human evaluation, we validate that our proposed framework exhibits the ability to generate datasets of higher quality compared to the baseline dialog inpainting model.

## introduction dialog systems @xcite are designed to engage in natural language conversations with users, provide relevant information, answer queries, and simulate human interactions. these systems have gained significant attention in both academics and industry owing to their various potential applications, such as online customer service, virtual assistants, and interactive chatbots @xcite @xcite . however, the scarcity of datasets poses a major challenge in * equal contribution. ## dialogizer dialogizer is a novel framework designed to generate contextually relevant convqa datasets of high quality from textual sources. in addition to the simple dialog reconstruction (dr) task ( §3.1), the framework incorporates two novel training methodologies: question-answer matching (qam) ( §3.2) and topic-aware dialog generation (tdg) ( §3.3). during the inference phase, the textual passage is segmented into sentences that serve as answers, and the trained model autoregressively generates questions relevant to each answer to complete the con-vqa dataset ( §3.4). furthermore, to consistently generate stable and high-quality questions, dialo-gizer employs re-ranking through beam search during the inference phase, taking into account the contextual relevance of the generated questions ( §3.5). figure 2 provides an illustrative overview of our framework. ## experiments we deploy the dialogizer framework to generate four datasets, validating the quality of the datasets through diverse evaluations. the experimental details are provided in appendix c. ## analysis 5.1 ablation study to gain deeper insight into our method, we conduct an ablation study comparing the quality of datasets generated by applying each methodology to wikipedia. as shown in question type proportion concept asking for a definition or explanation of a concept 38.49% example asking for a examples of a concept 32.91% verification seeking confirmation regarding truthfulness of a concept 15.54% judgemental requesting the answerer's own opinions 8.57% comparison asking for comparison between multiple concepts 4.49% for instance, the rqgue and usr-dr metrics, which evaluate relevance by considering context, questions, and answers, indicate that tdg yields greater performance enhancement. in contrast, qrelscore, which evaluates question-answer pairs, demonstrates the effectiveness of qam. the results obtained from training with both tdg and qam simultaneously indicate that the two approaches synergistically contribute to generating more contextually relevant data. furthermore, when performing re-ranking based on rqgue scores during inference, performance improvements are observed across all scenarios and metrics. ## conclusion this study proposes dialogizer, a framework that enables the automatic generation of high-quality dialog datasets from textual sources. dialogizer is trained with the tasks of dialog reconstruction, question-answer matching for contextual alignment, and topic-aware dialog generation for spe-cific question creation. our experimental results demonstrate that dialogizer produces contextually relevant convqa datasets. the proposed framework holds promise for advancing convqa research and its practical applications.