An Emotion-based K orean Multimodal Empathetic Dialogue System (2022)

Abstract: We propose a Korean multimodal dialogue system targeting emotion-based empathetic dialogues because most research in this field has been conducted in a few languages such as English and Japanese and in certain circumstances. Our dialogue system consists of an emotion detector, an empathetic response generator, a monitoring interface, a voice activity detector, a speech recognizer, a speech synthesizer, a gesture classification, and several controllers to provide both multimodality and empathy during a conversation between a human and a machine. For comparisons across visual influence on users, our dialogue system contains two versions of the user interface, a cat face-based user interface and an avatar-based user interface. We evaluated our dialogue system by investigating the dialogues in text and the average mean opinion scores under three different visual conditions, no visual, the cat face-based, and the avatar-based expressions. The experimental results stand for the importance of adequate visual expressions according to user utterances.

## introduction as dialogue systems for human-machine conversations have attracted attention from the public, various multimodal dialogue systems with the purpose of healthcare @xcite , empathetic conversation @xcite or multiparty attentive listening @xcite have been recently introduced because multimodality makes conversations more entertaining @xcite . most research in this field has been conducted by few research groups in industry or university because of the complicated architecture inherent in multimodal dialogue systems to control multimodal recognition or representation. consequently, most multimodal dialogue systems are *equal contribution limited to a few languages such as english and japanese. empathy is also the main factor for more humanized conversation @xcite along with multimodality. researches on empathetic dialogues @xcite @xcite @xcite are also focused on a few languages from a lack of empathetic dialogue datasets. although a korean empathetic dataset @xcite and a korean empathetic dialogue generation model @xcite have been recently published, a korean empathetic dialogue system supporting multimodality has not been studied. this paper makes the following contributions: the remainder of this paper is formed as follows. we explain our emotion-based korean multimodal empathetic dialogue system in section 2. in section 3, the experimental results of our dialogue system are discussed. section 4 contains the related work in multimodal dialogue systems and empathetic dialogues. finally, we draw our conclusion in section 5. ## empathetic dialogue system we illustrate the emotion-based korean multimodal empathetic dialogue system. as shown in fig. 1 , the overall architecture of the dialogue system is composed of modules on a device and server(s). the device must be equipped with at least a microphone, a speaker, a display, and a computer for voice activity detection, speech recognition, speech synthesis, and visual expression. the visual expression is derived from either a cat face-based emotion expression (v1) or an avatar-based gesture expression (v2). the modules on server(s) are an emotion detector, an empathetic response generator, a monitoring service, and the main controller to receive inputs (user information and a user speech in text) from the device and to send outputs (a system response in text, a detected emotion class, and estimated probabilities of a user emotion and a system dialogue strategy) to the device. those modules can operate on the device instead of server(s) if the computing and memory resources on the device afford them. otherwise, they can be executed on a single server or several servers in consideration of the resources on the server(s). ## experiments for evaluating our emotion-based korean multimodal empathetic dialogue system, we analyze the dialogue logs and the averaged mean opinion scores (mos) achieved by six participants. mos is commonly used to assess the dialogue system since no existing automatic evaluation metrics correctly measure the performance of the dialogue generation task. our dialogue system was also evaluated in three different visual-representing conditions which are no visual (a black screen), the cat face-based, and the avatar-based expression methods. ## related work several social robots providing multimodal interaction have been introduced for different purposes. the baby seal-shaped robot paro was developed by the national institute of advanced industrial science and technology in japan for robot therapy @xcite . and the paro robot was utilized for examining whether the robot can support family caregivers caring for older persons with dementia @xcite . the pepper robot, a wheeled humanoid robot produced by softbank robotics, was initially designed for business-to-business in softbank stores and has been utilized for a variety of applications for business-to-consumer, business-to-academics, and business-to-developers (pandey and gelin, 2018). @xcite created the erica robot, one of the most humanlike android robots, whose functionalities include conversation, advanced sensing, and speech synthesis. and the abilities of the er-ica robot extended into one-on-one attentive listening @xcite and multi-party attentive listening @xcite . the erica robot was also utilized for empathetic conversation during the covid-19 quarantine @xcite . as empathy plays a crucial role in communication, there have been several attempts to generate more empathetic system responses in text-based conversations. an end-to-end empathetic chatbot caire @xcite recognizes user emotions and generates responses in an empathetic manner, based on the generative pre-trained transformer @xcite . @xcite proposed a multi-factor hierarchical framework for empathetic response generation, which consists of communication mechanism, dialog act, and emotion. @xcite suggested a novel largescale dataset (pec) and a bert @xcite -based response selection model for personabased empathetic conversations. @xcite and @xcite focused on emotion causes for generating empathetic responses. @xcite leveraged commonsense to achieve additional information such as user's situations and feelings. and the information was utilized for the enhancement of empathetic response generation. ## conclusion this paper proposes an emotion-based korean multimodal empathetic dialogue system whose submodules include an emotion detector, an empathetic response generator, a monitoring interface, a web interface, and a unity interface. we evaluated our dialogue system by analyzing the dialogues in text and the average mean opinion scores under the three different visual-representing conditions and observed the significance of proper visual expressions. for future research, gesture classification with more specific-purpose gestures and system emotion expression corresponding to the system response will be considered.