S uper D ialseg: A Large-scale Dataset for Supervised Dialogue Segmentation (2023)

Abstract: Dialogue segmentation is a crucial task for dialogue systems allowing a better understanding of conversational texts. Despite recent progress in unsupervised dialogue segmentation methods, their performances are limited by the lack of explicit supervised signals for training. Furthermore, the precise definition of segmentation points in conversations still remains as a challenging problem, increasing the difficulty of collecting manual annotations. In this paper, we provide a feasible definition of dialogue segmentation points with the help of document-grounded dialogues and release a large-scale supervised dataset called SuperDialseg, containing 9,478 dialogues based on two prevalent document-grounded dialogue corpora, and also inherit their useful dialogue-related annotations. Moreover, we provide a benchmark including 18 models across five categories for the dialogue segmentation task with several proper evaluation metrics. Empirical studies show that supervised learning is extremely effective in in-domain datasets and models trained on SuperDialseg can achieve good generalization ability on out-of-domain data. Additionally, we also conducted human verification on the test set and the Kappa score confirmed the quality of our automatically constructed dataset. We believe our work is an important step forward in the field of dialogue segmentation.

## introduction recently, the importance of dialogue segmentation for dialogue systems has been highlighted in academia @xcite @xcite and industry @xcite . dialogue segmentation aims to divide a dialogue into several segments according to the * corresponding author discussed topics, and therefore benefits the dialogue understanding for various dialogue-related tasks, including dialogue summarization @xcite @xcite , response generation @xcite , response selection @xcite , knowledge selection @xcite , and dialogue information extraction @xcite . when using large language models (llms) as dialogue systems (e.g., chatgpt foot_0 ), dialogue segmentation methods can also help us to save tokens during long-term conversations while minimizing information loss. figure 1 shows an example of dialogue segmentation. owing to the scarcity of supervised dialogue segmentation datasets, previous studies developed unsupervised methods @xcite @xcite but easily reached the ceiling. noting that models trained with largescale supervised text segmentation datasets are effective for segmenting documents @xcite @xcite , however, @xcite experimented that applying them directly on dialogues were inadequate because dialogue has its own characteristics. previous studies showed that deep learning (dl) models usually perform better with supervised training signals and converge faster @xcite . therefore, despite the expensive cost of manual annotation, previous works still attempted to collect some annotated datasets. however, these datasets are either small, containing only 300 ∼ 600 samples for training @xcite , or not opensourced, due to the privacy concerns @xcite and other issues (e.g., copyright issues from the industry @xcite ). therefore, a large and publicly available supervised dataset is urgently needed. in addition, some studies on spoken dialogue systems provided manual annota-figure 1 : a document-grounded dialogue example selected from the doc2dial dataset @xcite . different colors correspond to different dialogue segments. apart from the cost, another challenge for constructing a high-quality annotated dataset is the vague definition of segmentation points in dialogues. in contrast to documents, which have clear structures such as sections and paragraphs, segmenting dialogues is particularly confusing and difficult. existing annotated datasets generally suffer from disagreement among annotators @xcite , resulting in relatively low kappa scores 2 or boundary ambiguity @xcite . inspired by @xcite , we conjecture that the document-grounded dialogue system (dgds) @xcite ) can provide a criterion for defining the boundary of dialogue segments. specifically, in a document-grounded dialogue, each utterance is grounded on a piece of relevant knowledge within a document, illustrating what speakers concentrate on or think about when talking. this information can reflect the discussing topic for each utterance. based on this characteristic, we construct a supervised dialogue segmentation dataset, superdialseg, based on two dgds datasets, containing 9,478 dialogues. moreover, we inherit their rich dialoguerelated annotations for broader applications. usually, dialogue segmentation is considered as a subtask that aids other downstream tasks, and thus, it lacks a standard and comprehensive benchmark for comparison. therefore, in addition to proposing the supervised dataset, we conducted an extensive comparison including 18 models across five categories using appropriate evaluation metrics. • we constructed a large-scale supervised dialogue segmentation dataset, superdialseg, from two dgds corpora, inheriting useful dialogue-related annotations. ## the superdialseg dataset documents have natural structures, whereas natural structures for dialogues are hard to define. in this paper, we utilize the well-defined document structure to define the dialogue structure. ## task definition dialogue segmentation aims to segment a dialogue @xmath0, u 2 , ..., u n } into several pieces according to their discussed topics, where u i contains not only the utterance u i , but also some properties including the role r i and the dialogue act da i . with the superdialseg dataset, we define it as a supervised learning task by labeling each utterance with @xmath1, otherwise, @xmath2. tiage @xcite is a manually annotated dataset extended from personachat @xcite comprising 500 samples, where 300, 100, and 100 samples are for the training, testing, and validation sets, respectively. dialseg711 @xcite is an evaluation dataset containing 711 multi-turn conversations, which are synthesized by combining singletopic dialogues sampled from the multiwoz @xcite and the stanford dialogue dataset @xcite . ## conclusions in this paper, we addressed the necessities and challenges of building a large-scale supervised dataset for dialogue segmentation. based on the characteristics of dgds, we constructed a large-scale supervised dataset called superdialseg, also inheriting rich annotations such as role information and dialogue act. moreover, we confirmed the quality of our automatically constructed dataset through human verification. we then provided an extensive benchmark including 18 models across five categories on three dialogue segmentation datasets, including superdialseg, tiage, and dialseg711. to evidence the significance and superiority of our proposed supervised dataset, we conducted extensive analysis in various aspects. empirical studies showed that supervised signals are essential for dialogue segmentation and models trained on superdialseg can also be generalized to out-ofdomain datasets. based on the comparison between the existing dataset (i.e. tiage) and superdialseg on the same scale, we observed that models trained on superdialseg had a steady training process and performed well on all datasets, probably due to its diversity in dialogue segmentation patterns, therefore, it is suitable for this task. furthermore, the comparison between roberta and mvroberta highlighted the importance of dialogue-related annotations previously overlooked in this task. additionally, we also provided some insights for dialogue segmentation, helping the future development in this field. moreover, our dataset and codes are publicly available to support future research. 6 limitations in this study, we notice that our proposed method of introducing dialogue-related information including role information and dialogue act (i.e., mvroberta) performs well but the way of exploiting these annotations is simple. we believe there is much room for improvement. therefore, we need to find a more proper and effective way to make full use of the dialogue-related information, expanding the potential applications of supervised dialogue segmentation models. besides, our dataset is only available for training and evaluating english dialogue segmentation models. recently, @xcite proposed a chinese dgds dataset with fine-grained grounding reference annotations, which can also be used to construct another supervised dataset for chinese dialogue segmentation. we leave it as a future work.