A Multimodal Dialogue System to Lead Consensus Building with Emotion-Displaying (2024)

Abstract: The evolution of large language models has enabled fluent dialogue, increasing interest in the coexistence of humans and avatars. An essential aspect of achieving this coexistence involves developing sophisticated dialogue systems that can influence user behavior. In this background, we propose an effective multimodal dialogue system designed to promote consensus building with humans. Our system employs a slot-filling strategy to guide discussions and attempts to influence users with suggestions through emotional expression and intent conveyance via its avatar. These innovations have resulted in our system achieving the highest performance in a competition evaluating consensus building between humans and dialogue systems. We hope that our research will promote further discussion on the development of dialogue systems that enhance consensus building in human collaboration.

## introduction the emergence of large language models, such as @xcite , has facilitated highly fluent text-based conversations. nevertheless, in many practical situations, dialogue systems require the capability to influence users through negotiation, persuasion, and consensus building @xcite . in these advanced dialogue scenarios, it is essential to impact users' cognitive and emotional responses to induce alternation in their thoughts, opinions, and behaviors, yet research on these skills remains limited @xcite . in this context, the dialogue system live competition 6 (dslc6) was organized to evaluate the current technological limitations and identify necessary components for creating systems capable of consensus building @xcite . the competition set up a dialogue scenario wherein the system had to negotiate with the user in the context of conflicting goals: a system and a user jointly plan a party, but the system desires a grand party while the user actually prefers a modest one. the systems are required to take the user's voice utterance as input and respond with avatar movements and synthesized voice. the demonstrations are evaluated based on three criteria to ensure they are contextually appropriate: relevance of utterance content, suitability of gestures and facial expression, and appropriateness of pause and voice modulation. this paper presents our system foot_0 submitted to dslc6 @xcite , which aims to build consensus with a user by guiding discussion with subdivided topics and conveying emotions and positions through its avatar. our focus in designing the consensus building process is on agenda management and the clear communication of emotions and positions. in order to facilitate discussion between speakers with different objectives, our system employs a strategy of dividing a large topic into subtopics and guiding the user step by step. specifically, to manage discussion flow, we predetermine a list of subtopics as "blanked slots" in the gpt-based system's prompt. once a particular issue is resolved, the system introduces the next unresolved subtopic. within each subtopic, negotiations progress through a sequence of proposals and responses (including acceptance, rejection, and counter-proposals) @xcite . throughout this process, the system is designed to articulate its intentions while responding. one crucial element in this effort is the expression of emotions, which is considered to influence the future actions of others in negotiations @xcite @xcite . when controlling the avatar, we incorporate facial expressions, voice modulation, and body poses to express emotions. through these innovations, our system demonstrated the best performance in dslc6, and stt then, it might be a good choice. the teacher likes the sushi there! why do you prefer the sushi bar? ⁝ ×6 dialogue history system prompt • basic information: • situation, agenda, intended goal • candidate labels with utterance • slot states: venue access budget price content scale sushi bar none none none none none *joy* should i make the reservation? emotion label utterance joy pose label none system should i make the reservation? response-generation module ( §3) avatar-control module ( §4) multimodal software desires a modest party should i make the reservation? update user voice ( §4.1) speed volume pitch 125 200 120 facial expression ( §4.2) … … pose ( §4.3) left_hand_palmup desires a grand party feedback from the evaluators confirmed their effectiveness. ## system overview our system consists of two modules: the responsegeneration module and the avatar-control module (figure 1 ). the response-generation module (section 3) generates the system's utterance and emotion/pose labels in response to the user's utterance recognized by a speech-to-text (stt) function. the generated responses are forwarded to the avatar-control module (section 4) that controls the voice, facial expressions, and poses according to predefined rules based on the received labels. ## response-generation module for the response-generation module, we employed gpt-4 to generate outputs. we provided the module with a system prompt including basic information regarding the dialogue setting and the current status of slots (section 3.1). subsequently, the module generated an utterance and corresponding emotion/pose labels based on the provided prompt. additionally, we ensured smooth turn-taking through pre-and post-module processing (section 3.2). ## avatar-control module in the avatar-control module, the system's emotions and position are effectively conveyed by reflecting rules corresponding to emotion/pose labels or utterance content in the avatar's voice, facial expressions, and poses. the avatar utilizes the resources provided by dslc6 @xcite . ## human evaluation three top-performing systems, selected from an initial pool of ten systems in the preliminary round, participated in the final round of dslc6. at the final round, each system engaged in a five-minute conversation in japanese with a user who is a humanities researcher. the interactions were evaluated by an audience of 80 attendees. each system was evaluated twice. the assessments were on the basis of contextual appropriateness, considering the following three criteria: (i) content: relevance of utterance content, (ii) expressions: suitability of gestures and facial expressions, and (iii) voice: appropriateness of pause and voice modulation. using a 5-point likert scale, the systems' performances were ranked based on the average scores of all criteria. the results in ## conclusion this paper presented the top-performing system in a dialogue competition focused on the consensus building process between systems and humans. our proposed system incorporated two strategies for smooth consensus building: topic control through a slot-filling approach and conveying intent through emotional expression via an avatar. our system can be adapted for various situations. the only task-specific components are the prompts and slots. the prompts include a general instruction format necessary for the consensus building process, such as context, personas, subtopics, and the system's objectives, thereby reducing the effort required for prompt engineering when applied to different tasks. each topic slot is managed by a separate model, allowing easy instantiation and repurposing once the subtopics are defined. we hope that this system will contribute insights to research on dialogue systems to build consensus with humans in the context of conflicting goals.