MODDP : A Multi-modal Open-domain C hinese Dataset for Dialogue Discourse Parsing (2024)

Abstract: Dialogue discourse parsing (DDP) aims to capture the relations between utterances in the dialogue. In everyday real-world scenarios, dialogues are typically multi-modal and cover open-domain topics. However, most existing widely used benchmark datasets for DDP contain only textual modality and are domain-specific. This makes it challenging to accurately and comprehensively understand the dialogue without multi-modal clues, and prevents them from capturing the discourse structures of the more prevalent daily conversations. This paper proposes MODDP, the first multi-modal Chinese discourse parsing dataset derived from open-domain daily dialogues, consisting 864 dialogues and 18,114 utterances, accompanied by 12.7 hours of video clips. We present a simple yet effective benchmark approach for multi-modal DDP. Through extensive experiments, we present several benchmark results based on MODDP. The significant improvement in performance from introducing multi-modalities into the original textual unimodal DDP model demonstrates the necessity of integrating multi-modalities into DDP.

## introduction dialogue discourse parsing (ddp) aims to identify the discourse relations between utterances in a dialogue using a dependency tree. the left part of figure 1 shows an example, where the arcs represent the dependencies between utterances and the labels on arcs are discourse relations. as a fundamental task in natural language processing (nlp), ddp can contribute to a deeper understanding of the structure and semantics inherent in dialogues, and has been proven beneficial for various downstream tasks, including emotion recognition @xcite , dialogue response generation @xcite 2020), and meeting summarization @xcite . to support ddp research, high-quality labeled data is indispensable. to date, stac @xcite and molweni @xcite are two existing publicly available benchmark datasets that widely used in prior research @xcite @xcite , promoting progress in the field of ddp. both of the two datasets are collected from dialogues in english-language online forums, where stac originates from the chat dialogues of an online game, and molweni is sourced from an online forum about ubuntu. although the dialogue scenarios for the two sources actually contain multimodalities beyond text, such as game graphics for the stac source and screenshots of the ubuntu interface for the molweni source, these datasets are focused solely on textual modality, excluding the existing useful related multi-modal information. consequently, based on these two datasets, most of the ddp works only rely on the textual modality to parse discourse structures @xcite @xcite . however, despite significant progress in existing ddp research, the neglect of valuable multimodal information for ddp still presents two challenges. first, conducting ddp within a single textual modality does not align with the real-world dialogue scenarios, since communication in real life often involves multiple modalities beyond text, such as visual and audio modalities. second, solely relying on the textual modality for ddp may limit the ability to achieve a comprehensive and accurate understanding of the dialogue. taking figure 1 as an example, from the text only, it is difficult to understand the dialogue "你...！"，"六十万". when referring to the angry tone of the person named chuan from audio modality, and his action of snatching the walnut from ayi's hand according to visual modality, it becomes clear that chuan is accusing ayi, while ayi warns chuan about the walnut's value of six hundred thousand yuan. this illustrates that audio and visual modalities can provide essential supplementary cues for ddp that text alone cannot convey. @xcite pioneered the introduction of multi-modalities to ddp, namely jddc2.1, consisting of dialogues from a mainstream chinese e-commerce platform. however, jddc2.1 still mainly focuses on the textual modality, since each dialogue contains an average of only two images, and thus more helpful multi-modal information, such as tones in audio and actions in video, cannot be fully exploited for ddp. moreover, all the above existing datasets originate from task-specific domains, thereby limiting their abilities to reflect the dialogue discourse structures of the more natural and widespread opendomain real-world scenarios. to address these limitations, this work proposes moddp, the first multi-modal open-domain chinese datasets for dialogue discourse parsing to the best of our knowledge, consisting of 864 twoparty dialogues, and 18,114 utterances, with parallel video clips of 12.68 hours. overall, moddp has the following important features. first, it is sourced from open-domain dialogues across various tv series, containing textual, audio, and visual modalities simultaneously. this capability allows it to more effectively reflect discourse structures in multi-modal daily dialogues, thereby facilitating research in multi-modal ddp to better support practical applications in daily life. second, we adopt the annotation scheme of sdrt (segmented discourse representation theory) @xcite following stac and molweni, capturing the whole discourse structure of a dialogue as a dependency tree, with 16 labels to distinguish discourse relations. we also compile comprehensive annotation guidelines for annotators' reference. third, to ensure data quality, we perform strict double annotation workflow. each dialogue is assigned to two annotators to independently annotate discourse structures, with a third expert annotator to handle inconsistency annotations. to provide benchmark results on our newly annotated moddp, we propose a simple yet effective benchmark approach for multi-modal ddp. we conduct extensive experiments and provide several benchmark results based on moddp, under both settings of using textual modality and using multi-modalities. experimental results and further analysis show that the ddp performance increase by large margin after introducing multi-modalities, highlighting the substantial benefits of integrating multi-modalities for ddp. we will release moddp datasets, along with our compiled annotation guidelines and codes for research usage at https://github.com/suda-iainlp/ moddp . stac (nicholas asher and afantenos, 2016) is collected from chat dialogues of an online game and adopts the annotation scheme of sdrt (lascarides and asher, 2009), with 16 labels specifically designed to capture the discourse relations for dialogues. following the annotation scheme of stac, @xcite construct a larger dialogue discourse structure dataset called molweni, deriving from the multiparty dialogues dataset the ubuntu dialogue corpus @xcite . gum @xcite also contains discourse annotations on dialogues. in their annotation schema, they adopt the rhetorical structure theory (rst) @xcite to capture the dis-course structure of dialogues. though contribute to the progress for ddp field, above datasets contain only textual modality, overlooking the prevalent multi-modalities in real-world scenarios that could help understand dialogue better. the first multi-modal ddp dataset is constructed by @xcite , namely jddc 2.1, which is derived from the online dialogues in a chinese e-commerce platform and also adopt sdrt annotation scheme. each dialogue in jddc2.1 contains multiple text utterances and two image utterances in average. as pioneered work for multi-modal ddp, jddc 2.1 can serve as a valuable benchmark dataset. however, each dialogue in jddc2.1 still mainly present by texts with few images, without audio or video modalities. besides, as a taskspecific domain dataset, it is not able to reflect the dialogue discourse structure in the more natural and prevalent multi-modal open-domain real-world scenarios. therefore, in this paper, we propose the first high-quality multi-modal chinese discourse structure dataset for open-domain daily dialogues, with parallel text, visual, and audio modalities simultaneously, aiming to facilitate comprehensive research in the field of open-domain daily dialogues. ## data construction in this section, we describe the annotation methodology and annotation procedure in detail. data selection. in order to build a multi-modal open-domain ddp dataset that can illustrate the discourse structure in real-world daily conversations, we follow the work of @xcite on multi-modal emotion recognition in two-party conversations, and select their collected video dialogue clips from several different chinese tv series with the categories of family, romance, soap opera, and modern opera. finally, we obtain 864 dialogues, with a total of 18,114 utterances for annotation. each utterance in the dialogue contains both text and its aligned video clip. annotation guideline. after comparing three mainstream discourse formalisms, including penn discourse treebank (pdtb) @xcite , rhetorical structure theory (rst) @xcite , and segmented discourse representation theory (sdrt) @xcite , we adopt the sdrt-style annotation guideline defined in stac dataset (nicholas asher and afantenos, 2016) based on the following considerations. compared with pdtb that focus on shallow discourse relations between a pair of utterances without considering the whole discourse structure, and rst which can capture the whole discourse structure but does not allow non-adjacent relations, sdrt is able to represent the overall discourse structures of dialogues and handle the non-adjacent relations that can occur in dialogues. moreover, the sdrt-style annotation guideline released by stac is specifically designed for dialogues, with 16 labels to distinguish discourse relations as shown in table1, and gives detailed illustrations and examples to ensure annotation quality. quality control. we employ 7 postgraduate students as our part-time annotators, and select 2 capable annotators with linguistic background as expert annotators to deal with annotation inconsistency. we compile a comprehensive annotation guideline for annotators' reference, which we upload as supplementary materials. the annotators are asked to annotate all the utterances in the dialogue sequentially with reference to the corresponding video clip. the annotation process consists of three parts. first, given the text of an utterance, we ask the annotator to watch its corresponding video clip. second, they need to recognize a previous utterance that has discourse relation with the current utterance after watching the video. finally, they are asked to identify the relation type between the two discourse-related utterances from among the 16 labels in the annotation guideline. we build an annotation tool to support the above annotation process, as shown in appendix c. before formal annotation, each annotator is trained for several hours to be familiar with annotation guidelines and our developed annotation tool. during annotation, we perform strict double annotation based on our annotation tool to guarantee the quality of the labeled data. specifically, each dialog is randomly assigned to two different annotators to independently annotate the whole discourse structure. if the annotations submitted by two annotators are the same, we directly take the consistent annotation as the final answer, otherwise, a third expert annotator decides the final answer after analyzing the two inconsistent submissions. ## analysis on moddp in this section, we analyze our annotated data from different perspectives. inter-annotator consistency. we use cohen's kappa value @xcite to calculate the interannotator consistency in the above mentioned double annotation workflow. for the consistency on discourse dependency links, the kappa value is 0.95, which is a very high agreement because most of the dependency links occur between adjacent utterances and thus can be easily recognized. we will give more discussion on dependency distances below. for the consistency on both links and relations, the kappa value is 0.63, which is higher than that of in the ddp datasets with only textual modality, such as 0.56 in stac and 0.58 in molweni datasets. this demonstrates that the multi-modal information can help the annotators better determine discourse structures in dialogues, which we discuss in detail in appendixb. even with a higher kappa value attained, the relatively low consistency in both link and relation labels reflects the difficulty in distinguishing discourse relations. this demonstrates the importance of performing double annotation to ensure data quality. ## approach in this section, we present a simple yet effective benchmark approach for multi-modal open-domain ddp, aiming to provide reliable benchmark results on our newly annotated dataset to facilitate researchers in their further exploration. audio-encoder visual-encoder text-encoder multi-modal interaction module context-speaker-aware dialogue interaction module contextual block inter-speaker block intra-speaker block audio visual text k v q score 𝑗 → 𝑖 score 𝑗 → 𝑖 mlp link mlp rel ## conclusions this paper proposes moddp, the first multi-modal open-domain dialogue discourse parsing dataset, consisting of 864 dialogues and 18,114 utterances with aligned video clips of daily conversations. we present a detailed description of the construction process for moddp and give in-depth data analysis. we propose a simple yet effective benchmark approach for multi-modal ddp, and conduct ex-tensive experiments, providing several benchmark results under both textual unimodal setting and multi-modal setting based on our newly constructed moddp. experimental results demonstrate the significant benefits of multi-modalities for ddp. we hope that moddp will facilitate future research in the valuable yet under-explored field of multimodal ddp.