MP 2 D : An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs (2024)

Abstract: Despite advancements in on-topic dialogue systems, effectively managing topic shifts within dialogues remains a persistent challenge, largely attributed to the limited availability of training datasets. To address this issue, we propose Multi-Passage to Dialogue (MP2D), a data generation framework that automatically creates conversational question-answering datasets with natural topic transitions. By leveraging the relationships between entities in a knowledge graph, MP2D maps the flow of topics within a dialogue, effectively mirroring the dynamics of human conversation. It retrieves relevant passages corresponding to the topics and transforms them into dialogues through the passage-to-dialogue method. Through quantitative and qualitative experiments, we demonstrate MP2D’s efficacy in generating dialogue with natural topic shifts. Furthermore, this study introduces a novel benchmark for topic shift dialogues, TS-WikiDialog. Utilizing the dataset, we demonstrate that even Large Language Models (LLMs) struggle to handle topic shifts in dialogue effectively, and we showcase the performance improvements of models trained on datasets generated by MP2D across diverse topic shift dialogue tasks.

## introduction dialogue systems @xcite , designed to respond to inquiries or provide relevant information, have gained considerable attention in academia and industry. these systems show promise in various applications, including virtual assistants, customer service, and chatbots @xcite @xcite . nonetheless, while numerous studies have advanced the field of conversational systems within on-topic dialogue scenarios, significant challenges persist concerning dialogues encompassing topic shifts @xcite @xcite . unlike human conversation, which can naturally flow between topics and continue the discussion seamlessly, current dialogue systems often struggle to determine the appropriate timing for topic shifts or to execute these shifts fluently @xcite . a major challenge in these tasks lies in the scarcity of data. this issue is compounded by the fact that creating all existing topic shift dialogue datasets @xcite involves a laborious and costly human annotation process. to address this challenge, we propose a novel framework, multi-passage to dialogue (mp2d), specifically crafted for the automatic generation of conversational question-answering (convqa) data featuring topic shifts. the framework lever-ages the passage-to-dialogue (p2d) method @xcite in convqa dataset generation, which entails generating relevant questions by using the sentences within a passage as answers. unlike existing methods that construct dialogues from a single passage, our proposed framework integrates multiple passages to create dialogues with topic shifts. to emulate the dynamic nature of real-world conversations, we employ a knowledge graph to identify paths connecting various entities and their relations, as shown in figure 1 . by retrieving passages using entities in the path as queries, the passages and the relation sentences between entities form a multi-passage structure. this structure is then segmented into sentences, which directly serve as answers, and a question generator generates suitable questions to complete dialogues with natural topic transitions. we conduct experiments using various question generators to transform multi-passage into dialogue, employing diverse reference-free dialogue metrics for automatic evaluation. the results indicate that the generated topic shift dialogue datasets demonstrate high quality when utilizing large language models (llms) @xcite as a question generator within mp2d. moreover, through qualitative assessments conducted via human and gpt-4 evaluation @xcite , we found that approximately 91% of the generated dialogues are deemed excellent in their handling of topic shifts. furthermore, we introduce a novel topic shift dialogue benchmark, ts-wikidialog, to evaluate llms in handling topic shift dialogue tasks and to broaden the application of datasets generated via mp2d. the benchmark is constructed by paraphrasing the mp2d-generated dialogue by llms and human annotators. in experiments utilizing the ts-wikidialog, we initially verify that various llms struggle to effectively address problems related to topic segmentation @xcite , topic shift detection @xcite , and topic shift convqa @xcite . furthermore, the t5-base model @xcite , trained on the mp2d-generated dataset, exhibits significantly better performance in topic segmentation and topic shift detection tasks compared to baseline llms. this validates the mp2d framework as an effective dataset-creation approach for tackling topic shift challenges. additionally, we demonstrate that utilizing finetuned models on mp2dgenerated datasets for the topic shift detection task can enhance the performance of llm responses in convqa, especially in topic shift turns. ## multi-passage to dialogue (mp2d) the multi-passage to dialogue (mp2d) framework represents a novel approach tailored to generating dialogues with topic shifts by incorporating multiple textual passages. through the utilization of a knowledge graph, mp2d extracts paths to establish natural linkages among topics, facilitating smooth transitions within the dialogue. the framework retrieves passages by querying each entity within these paths, forming a multi-passage structure by pairing these passages with sentences that describe the relationships between entities ( §3.1). subsequently, it segments the multi-passage into sentence-sized units to serve as answers. leveraging the capability of llms to generate contextually appropriate questions for each answer, mp2d employs llms as question generators in crafting dialogues ( §3.2). an automatic post-processing phase follows, resulting in dialogues that seamlessly transition between topics. figure 2 provides an illustrative overview of the proposed framework. ## evaluating the mp2d framework in this section, we empirically demonstrate that the mp2d framework is capable of automatically generating high-quality dialogue with smooth topic shifts, both quantitatively ( §4.1) and qualitatively ( §4.2). first, we employ existing p2d methods and the llm as question generators to compare these approaches using various referencefree dialogue metrics. within the mp2d framework's passage retrieval component, we employ the wikipedia dataset to generate 10,000 multi-turn dialogues, subsequently evaluating their quality. furthermore, absolute evaluations are conducted using various criteria to illustrate that the generated topic shift dialogues exhibit a natural transition of topics. ## applications to illustrate the efficacy of mp2d, we evaluate the performance of models trained on datasets produced by the mp2d framework in two topic shift tasks: topic segmentation and topic shift detection. since mp2d is the first framework for automatically generating topic shift convqa data, we experiment with baseline models that include zeroshot and few-shot settings of llms, as well as models trained on dialogues constructed from randomly selected topics without the use of knowledge graphs (mp2d random-shift). our analysis aims to validate that the mp2d framework is an effective data generation tool, producing datasets that enable models to achieve high performance on diverse topic shift tasks. we generate 10,000 dialogues using the mp2d framework for training purposes and employ ts-wikidialog as the test set, ensuring there is no overlap in topics or retrieved passages between the training and test sets. further details, including the prompts for each task, can be found in appendix g. ## case study we present examples drawn from the mp2dgenerated topic shift dialogues in ## conclusion in this work, we address the data scarcity issue in topic shift dialogues by proposing a framework that automatically generates dialogues with natural topic transitions. our mp2d methodology utilizes the flow of relationships between entities in a knowledge graph to structure the dialogue flow and converts multi-passage content into a convqa format. experimentally, we demonstrate that mp2dgenerated topic shift dialogues are of high quality and prove their value as a training dataset for various topic shift dialogue tasks.