D 4 AC : A Tool for Developing Multimodal Dialogue Systems without Coding (2025)

Abstract: To enable the broader application of dialogue system technology across various fields, it is beneficial to empower individuals with limited programming experience to build dialogue systems. Domain experts, where dialogue system technology is highly relevant, may not necessarily possess expertise in information technology. This paper presents D4AC, which works as a client for text-based dialogue servers. By combining D4AC with a no-code tool for developing text-based dialogue servers, it is possible to build multimodal dialogue systems without coding. These systems can adapt to the user’s age, gender, emotions, and engagement levels obtained from their facial images. D4AC can be installed, launched, and configured without technical knowledge. D4AC was used in student projects at a university, which suggested the effectiveness of D4AC.

## introduction recent advancements in large language models have made it possible to develop various kinds of dialogue systems easily. in applying dialogue systems to various fields, it is effective for dialogue system engineers to cooperate with experts in the fields. however, there is a problem that the resources of dialogue system engineers are limited. therefore, it would be desirable that experts in those fields could construct a dialogue system without coding. although there are tools for building text-based dialogue systems without coding, there have been no no-code tools for building multimodal dialogue systems that can exploit the user's attributes and social signals @xcite obtained from the user's images. such systems are expected to engage in communication better than text-based systems, as they can adapt their communication strategies to the user's attributes and states. ## related work there are studies on dialogue systems that handle information obtained from the user's image. for example, there are studies on a dialogue system that recognizes affirmative/negative attitudes from prosody and head movements @xcite and a reception dialogue system that smoothly facilitates turn-taking by estimating user engagement from standing positions and recognizing turnyielding or taking intentions from the gaze and gestures @xcite ). an interview dialogue system that changes dialogue strategies by estimating the willingness to speak from gestures and speech @xcite is also studied. simsensei @xcite is a system that conducts interviews for ptsd diagnostic support, estimating the user's emotions from facial expressions, head movements, gaze, and prosody. while there are various studies in the field, there are few tools available as off-the-shelf solutions. '\psi' @xcite and virtual human toolkit @xcite are exceptions, but technical knowledge is required to use them. in the dialogue system live competition 6 @xcite , a tool that enables the development of dialogue systems using user state estimation results without coding was distributed to the participants, but, unlike d4ac, it can send the user states to the dialogue server only at the end of user utterances, and it is not publicly available. ## design of d4ac for user state estimation, d4ac uses face++. foot_3 if only estimating engagement, it is also possible to use facemesh of mediapipe. 5 for speech recognition and synthesis, it utilizes the web speech api of google chrome. alternatively, amazon polly foot_5 can be used for speech synthesis. d4ac can work as a client of dialogue servers that can be built without coding. it assumes ntt docomo's xaiml sunaba (sunaba hereafter) 7 and dialbb foot_7 (nakano and komatani, 2024) as tools for building and deploying dialogue servers. they provide gui-based editors for state transition networks to make it possible to write dialogue scenarios without coding. they also enable running dialogue servers as http servers. ## implementation here we explain the implementation details of d4ac according to figure 2 . the modules for user state estimation communicate with each other using mqtt, 10 which enables lightweight, asynchronous communication. the image capture module captures camera images and sends them to the mqtt broker. user state is estimated from these images using services like face++ and others. in the current version of d4ac, engagement is estimated using only the face directions obtained from face++ or mediapipe. we plan to replace this with a more accurate algorithm using facial expressions and gaze @xcite in the future. adding and replacing modules are easy thanks to the mqtt-based architecture. the dialogue server connection module sends the results of speech recognition and user state estimation to the dialogue server and returns the system utterance received in response to the main module. if the system utterance includes a userstate transmission request, it is remembered and used to decide whether to send the user state at the times of ( 2 ) and (3) in section 3.2. additionally, if specified, d4ac can send the system utterance text strings received from the dialogue server to an external auxiliary output server. by embedding commands within the system utterance text, it becomes possible to display images or to operate cg agents or robots. the frontend part is written in javascript, and other modules are written in python. ## sample application this section describes a simple example application. an example dialogue between this application and a user is shown in figure 7 . the dialogue server is built using dialbb. the scenario involves light conversations about food. in this scenario, chatgpt (gpt-4o foot_9 ) is used to generate system utterances in some states. the third, fourth, and sixth system utterances in figure 7 were generated by chatgpt. the configuration of the dialbb application includes the situations and persona of the system. speaker utterance (speech recognition result) remarks system hello i'm a chatbot. if you don't mind, could you tell me your name? user hello, i'm mike. (oh, i'm mike) system thank you mike! let me ask you about sandwich. do you have sandwiches very often? user yes, i like sandwiches very much. (yes i like sandwiches very much) system what's your favorite kind of sandwich, mike? i love a good blt, but i'm always looking for new ideas to try! the dialogue server requests d4ac to tell if the user engagement level is low or middle at the end of and two seconds after the system utterance. user i like chicken salad sandwiches. (i got chicken salad sandwiches) system chicken salad sandwiches sound delicious! do you make them yourself or do you have a favorite place to get them? the dialogue server requests d4ac to tell if the user engagement level is low or middle at the end of and two seconds after the system utterance. at the end of the system utterance, d4ac informs the dialogue server the user engagement level is low. system aren't you interested? the system asks the user if she/he is interested in the conversation. user oh, yes. i'm interested. sorry. (oh yes i'm interested sorry) system no worries, mike! do you add anything special to your chicken salad sandwiches, like grapes or nuts, or do you prefer to keep it simple? since the user response is affirmative, the system goes back to the conversation. (otherwise, the system wraps up the conversation.) at some states, the dialogue server requests d4ac to send the user's engagement level if it is "middle" or "low". then when the dialogue server receives the engagement level, it makes an utterance to ask if the user is bored with the conversation, and if so, the conversation is terminated. d4ac is configured to send the user's state to the dialogue server only when requested by the dialogue server. this occurs either at the end of the system's utterance or if the user does not speak for two seconds after the system's utterance has ended. the source files for this sample application are publicly available as a part of the d4ac repository. ## using d4ac in student projects during the development of d4ac, we utilized it in student projects at a university. the student projects were conducted twice, in the academic years 2021 and 2022, where students were tasked with building simple multimodal dialogue systems. four students participated in each project. in academic year 2021, the four students built separate dialogue systems following their interests. specifically, these were (1) a system for scheduling appointments, (2) a system for presenting quizzes, (3) a system explaining the use of shared cars, and (4) a system providing useful information for lgbtq+ individuals. the student who developed the fourth system was affiliated with a humanities research lab, while the others belonged to information technology or engineering labs. of these, systems (3) and ( 4 ) utilized d4ac along with sun-aba, but they did not use user state estimation results. at that time, d4ac could only send user state information at the end of user utterances, which limited its utility. following this experience, we modified d4ac to transmit user state estimation results at times other than the end of user utterances. in academic year 2022, the project involved four students working collaboratively to build a single system. among these students, two were from humanities and two were from information technology or engineering labs. they developed a dialogue system designed to calm the irritations of drivers while driving. the system converses with the driver, calming them if they are irritated, and the progression is determined based on the user state estimation results indicating whether the irritation is resolved. insights from humanities research (e.g., @xcite ) were utilized in creating the scenarios. this system used dialbb to build the dialogue server. this project showed that a system utilizing user state estimation results can be easily built with d4ac. we plan to provide opportunities for people from various fields to use the system and validate its effectiveness in future projects. ## concluding remarks this paper described d4ac, which is a tool that allows for the construction of multimodal dialogue systems without coding. it estimates user attributes and states from facial images and alters the course of dialogue based on these results. d4ac enables people from various fields, not just information technology, to build multimodal dialogue systems, contributing to the integration of dialogue system technology with other disciplines. although d4ac does not include novel technologies that improve empirical evaluation results, it concretizes a reasonable architecture for creating a multimodal dialogue system without coding. we believe that it will contribute to multimodal dialogue system technologies being used in various fields. we are considering the following enhancements and features. first, we aim to enable the use of tools and apis other than face++ and mediapipe for user state estimation. another goal is to integrate prosodic information from user utterances and text-based data for emotion estimation. additionally, we are considering the use of other user states such as sentiment @xcite and affirmative/negative attitudes @xcite . we also think that multimodal foundational models such as gpt-4o might be useful for user state estimation. as d4ac is released as an open source, we hope we will receive feedback for improvements while it is being used in an actual system development.