DSLCMM : A Multimodal Human-Machine Dialogue Corpus Built through Competitions (2025)

Abstract: A corpus of dialogues between multimodal systems and humans is indispensable for the development and improvement of such systems. However, there is a shortage of human-machine multimodal dialogue datasets, which hinders the widespread deployment of these systems in society. To address this issue, we construct a Japanese multimodal human-machine dialogue corpus, DSLCMM, by collecting and organizing data from the Dialogue System Live Competitions (DSLCs). This paper details the procedure for constructing the corpus and presents our analysis of the relationship between various dialogue features and evaluation scores provided by users.

## introduction with the advancement of large language models, the capabilities of text-based dialogue systems have improved @xcite @xcite . however, the performance of multimodal dialogue systems, which must process speech, facial expressions, and other non-verbal cues in real time, remains limited compared to human face-to-face interactions @xcite . to unlock the full potential of dialogue systems in society, it is essential to achieve multimodal dialogue capabilities akin to human interactions. however, there is a significant shortage of corpora to support this. while a relatively large amount of multimodal data focuses on human-to-human dialogue or human-to-wizard of oz (woz) interactions, there is a notable lack of corpora capturing dialogues between actual multimodal dialogue systems and humans. to understand how real systems and humans interact in multimodal dialogues and identify areas for improvement, a corpus of such interactions is indispensable. figure 1 : example data in dslcmm displayed using elan @xcite . in light of this background, we have constructed a multimodal dialogue corpus called dslcmm. specifically, we collected and processed data from the dialogue system live competition (dslc) series @xcite and organized it into a corpus. this corpus contains 1,747 dialogues between 32 multimodal dialogue systems and human users, obtained from two editions of the competition. the language of the corpus is japanese. in addition to users' speech, the corpus contains user/system video recordings and logs of system commands for gestures and facial expressions. it also includes subjective evaluation scores from users and transcriptions of all user utterances. an example of the dataset is shown in figure 1 . utilizing this dataset has the potential to significantly advance research on multimodal dialogue systems. the corpus will be accessible from the project pages of dslc5 foot_0 and dslc6 foot_1 . in section 2 of this paper, we review related work. section 3 provides an explanation of the dslc series, which served as the source of data for our corpus, and section 4 describes the dataset construction process and presents statistical information. in section 5, we discuss the analyses conducted on this corpus and the corresponding results. a brief summary is provided in section 6, followed by a discussion of limitations and ethical considerations. ## related work to the best of our knowledge, there are few existing datasets of dialogues between humans and multimodal dialogue systems. however, several datasets are available for multimodal dialogues between humans or between humans and a woz system. for example, meld @xcite includes video data from the tv series "friends", with annotations for emotions. additionally, multimodal datasets specifically focused on emotions, such as iemocap @xcite and mosi @xcite , have been constructed. the d64 multimodal conversational corpus @xcite contains data from natural conversations between humans collected using cameras and motion capture devices. ami @xcite is a corpus of meetings that includes video recordings of discussions. cejc @xcite captures everyday conversations and contains video data of human-to-human interactions across various daily activities. one notable multimodal dialogue dataset between humans and a woz system is the hazumi corpus @xcite . this corpus contains casual conversations between humans and a multimodal dialogue system operated by a wizard, along with subjective evaluation scores from users. analyses examining the relationship between system behavior and user evaluations have been conducted @xcite . however, since the corpus does not include dialogues between humans and autonomous dialogue systems, it is limited in addressing the challenges associated with developing and improving real multimodal dialogue systems. it should be noted that the term "multimodal dialogue" can also refer to dialogues discussing visual or video contents. well-known examples include mmconv @xcite , videochat @xcite , and simmc @xcite . however, these deal with text-based chat systems that interact with images, videos, or virtual reality environments, and do not involve face-to-face interactions typical of human dialogues, which are the focus of this study. ## dialogue system live competitions the dslc is a competition for dialogue systems that has been held in japan since 2018 @xcite . dslc consists of preliminary and final rounds, with the final round featuring a live event where dialogue systems are demonstrated in front of an audience, and rankings are determined based on audience evaluations. initially, the competition focused solely on text-based dialogue systems, but starting with dslc5 in 2022, it expanded to include multimodal dialogues @xcite . in this section, we describe dslc5 and dslc6, from which the data for our corpus were sourced. since the dataset is created from the preliminary round data, the final round is not discussed in this paper. ## corpus as the organizers of the dslc5 and dslc6, we processed the data of the preliminary rounds to create a multimodal dialogue corpus between users and multimodal dialogue systems. specifically, we extracted video segments for each dialogue session, transcribed the audio (transcriptions were performed only for user utterances, as the system's utterances were logged), and linked these with system logs and subjective evaluation scores. the data for each dialogue session consists of an elan the distribution of subjective evaluation scores for each subset is presented in ## analyses in lc5o, significant positive correlations were observed between the number of system utterances, the number of gesture and facial expression commands, and the evaluation scores. this suggests that the system made a good impression on users by providing informative utterances and expressing gestures. regarding latency, a positive correlation was observed with topic following. this indicates that longer response times were associated with better subjective evaluations, potentially because longer response times led to higher-quality responses regarding topics. in lc5s, a significant negative correlation was observed between the number of facial expression commands per utterance and the evaluation score. this suggests that expressing appropriate facial expressions to match specific situations may be more challenging compared to the open track. in lc6s, a negative correlation was found with the number of user utterances, suggesting that systems requiring users to speak extensively were likely more difficult to interact with. similar to lc5o, the number of gesture commands showed a positive correlation. additionally, the evaluation score exhibited a positive correlation with the number of facial expression commands. here, an inverse correlation relative to the lc5s results was observed, suggesting that further detailed analyses of the specific types of expressions and their contextual circumstances are needed. the analysis presented here is based on overall trends observed across multiple systems, and such insights could not be obtained from a dataset featur- ## summary in this study, we utilized data from the dslc series to construct dslcmm, a multimodal humanmachine dialogue corpus. dslcmm encompasses dialogues from multiple systems, capturing a wide range of dialogue quality. it includes data on system gestures and facial expression commands, as well as video data featuring facial information of users, making it well-suited for tasks such as multimodal dialogue generation. with user evaluation scores included, the dataset can also support tasks like estimating user evaluation scores @xcite , detecting dialogue breakdowns in multimodal settings @xcite @xcite . it can also be useful for building multimodal models for faceto-face conversation @xcite . we hope this corpus will advance research and development in multimodal dialogue systems.