Key Challenges in Multimodal Task-Oriented Dialogue Systems: Insights from a Large Competition-Based Dataset (2025)

Abstract: Challenges in multimodal task-oriented dialogue between humans and systems, particularly those involving audio and visual interactions, have not been sufficiently explored or shared, forcing researchers to define improvement directions individually without a clearly shared roadmap. To address these challenges, we organized a competition for multimodal task-oriented dialogue systems and constructed a large competition-based dataset of 1,865 minutes of Japanese task-oriented dialogues. This dataset includes audio and visual interactions between diverse systems and human participants. After analyzing system behaviors identified as problematic by the human participants in questionnaire surveys and notable methods employed by the participating teams, we identified key challenges in multimodal task-oriented dialogue systems and discussed potential directions for overcoming these challenges.

## introduction task-oriented dialogue systems are in high demand in both academia and industry @xcite . in recent years, particular attention has been given to multimodal task-oriented dialogue systems that enable more natural and richer real-time interactions through spoken utterances and visual representations of both participants (i.e., real-time camera input capturing the user, as well as output provided via system avatar video or embodied robot actions) @xcite . in this paper, we refer to these dialogues utilizing both audio and visual modalities as "audio-visual dialogues." despite significant attention in this area, the challenges involved in audio-visual task-oriented dialogue between humans and systems have not been sufficiently explored or shared within the research community. consequently, researchers are forced to define improvement directions individually without a clearly shared roadmap. this situation is mainly due to the lack of datasets for audio-visual task-oriented dialogues between humans and systems. without detailed analyses of such datasets, it is difficult to precisely identify the key factors influencing user experience and task performance, particularly given the complex interactions across multiple modalities. as an initial step toward promoting a shared understanding of key challenges in audio-visual task-oriented dialogues, we organized a competition at the 7th dialogue system live competition (dslc7) @xcite in which we constructed a dataset of japanese audio-visual task-oriented dialogues between various participating systems and humans, as shown in fig. 1 . this competition consisted of two stages: a preliminary round and a final round. in the preliminary round, 94 human evaluators interacted with the participating systems and provided subjective evaluations after each dialogue. as a result, we collected 1,865 minutes of publicly shareable data comprising 257 audio-visual task-oriented dialogues along with their corresponding subjective evaluation results. in the final round, the top three systems selected from the preliminary evaluations engaged in a total of six dialogues with the representatives we invited. the six dialogues were observed and evaluated by an audience of approximately 110 people, including dialogue researchers. additionally, we surveyed the audience before and after the final round, enabling us to analyze changes in their perceptions regarding key challenges in audio-visual dialogue systems throughout the competition. in this paper, we first describe the competition we organized and the dataset created during the competition. we then analyze the competitionbased dataset to identify key challenges in multimodal task-oriented dialogue systems involving audio and visual modalities. finally, we discuss potential approaches to address these challenges based on notable methods implemented by participating teams. our dataset and the codebase of system templates provided to participating teams (section 3.1) will be accessible from the dslc7 project page. 1 2 related work multimodal task-oriented dialogue datasets. various multimodal task-oriented dialogue datasets have been developed in response to the growing interest in multimodal dialogue. however, these datasets generally consist of dialogues that lack audio (i.e., spoken utterances) or visual modalities (i.e., videos of dialogue participants) @xcite @xcite @xcite or that have been collected either from human-human interactions or via the wizard-of-oz approach @xcite @xcite @xcite @xcite @xcite @xcite 1 https://sites.google.com/view/dslc7 . @xcite @xcite @xcite @xcite @xcite . consequently, the analysis of audio-visual multimodal task-oriented dialogues between actual systems and human users remains limited. in this study, we constructed an audiovisual task-oriented dialogue dataset containing interactions between diverse systems and diverse human users through organizing a competition. ## organization of competition as a first step towards identifying and sharing key challenges in audio-visual task-based dialogue systems, we organized a competition at dslc7 in japan. through this competition, we collected japanese dialogues between participating systems and humans and associated evaluation data. constructing a dataset through a competition enabled us to collect dialogues of diverse systems. the competition consisted of preliminary and final rounds. in the preliminary round, dialogue systems from participating teams engaged in faceto-face interactions with human evaluators. after each dialogue, human evaluators provided subjective evaluations of the systems. systems that received high subjective evaluation scores advanced to the final round. in the final round, these selected dialogue systems interacted with dialogue interlocutors designated by us in front of an audience that included dialogue system researchers. the audience evaluated the systems based on the demonstrated dialogues as a third party. ## collected data preliminary round data. we collected 257 dialogues between nine systems and 94 human evaluators, totaling 1,865 minutes, along with subjective evaluation results provided by evaluators. each dialogue consists of a quadruple: a frontal video of the interlocutor, a video of the dialogue system avatar and the travel viewer, separated audio recordings of both dialogue speakers, and system input/output logs. to the best of our knowledge, this is the first large-scale dataset of audio-visual task-oriented dialogues between humans and systems. more detailed information on the dataset is provided in appendix c. ## analysis we collected a large-scale dataset comprising audio-visual task-oriented dialogues between diverse dialogue systems and human evaluators and corresponding evaluation results. by analyzing them, we investigate the current key challenges in audio-visual task-oriented dialogues. numerous analytical perspectives are possible in audio-visual task-oriented dialogues, where multiple modalities intricately interact to influence user experience and task accomplishment. in this paper, we focus on the issues that were frequently pointed out in the preliminary and final round questionnaires. these issues may have a substantial impact on the dialogues, as they drew notable attention from evaluators in the preliminary round and from the audience in the final round. given that our dataset represents the first large-scale collection of audio-visual task-oriented dialogues between humans and systems, our analysis specifically emphasizes the issues associated with the audio-visual modality rather than the semantic content of the dialogues. ## conclusion in this paper, we presented an overview of the audio-visual task-oriented dialogue system competition we organized and the dataset constructed through this competition. to the best of our knowledge, this is the first publicly available large-scale dataset of audio-visual task-oriented dialogues between humans and systems. based on feedback from evaluators, we identified key challenges in developing multimodal task-oriented dialogue systems that integrate both auditory and visual modalities. our analysis further revealed that addressing some of the key challenges requires more than a direct application of existing research findingsit necessitates extending these insights, taking the nature of the task into account. we hope that the dataset and findings presented in this paper will contribute to more systematic and efficient research and development of audio-visual task-oriented dialogue systems.