E mpathy E ar: An Open-source Avatar Multimodal Empathetic Chatbot (2024)

Abstract: This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access.

## introduction the artificial intelligence (ai) community has witnessed significant progress in recent one year due to the explosive development of large language models (llms; @xcite , leading to unprecedented levels of intelligence in current ai systems. it is also a longstanding consensus that achieving human-level ai necessitates not only intelligence but also the capability to emulate human emotions, such as understanding feelings and perspectives and exhibiting empathy. the task of erg @xcite has then been developed with the aim of enabling machines to generate replies to user queries that are not only problem-solving but also emotionally inclined and empathetic, thereby facilitating emotionaware open-domain dialogues. erg serves as an effective testbed of machines' emotional intelligence, supporting emotional interactions with humans, and has been applied in various practical scenarios, e.g., mental health therapy and companion dialogue systems. however, current erg systems are significantly limited by their reliance on a single text modality in task definitions. emotional nuances are often more fully expressed and understood through nontext modalities in many scenarios, suggesting a gap in the current research. it's intuitive that, in many cases, human emotions are more effectively conveyed and perceived through vocal cues (such as the tone and pitch of speech), and/or dynamic visual changes in expressions (such as facial microexpressions and gestures), rather than through text alone. in contrast, relying solely on text responses from machines could never achieve the full spectrum of emotional resonance and empathy that human interactions offer. similarly, users may prefer to express their emotions through speech or facial videos, rather than being confined to text queries. regrettably, to date not much research has been carried out on the generation of multimodal empathetic responses from multimodal inputs. to fill this gap, this work is dedicated to developing a novel multimodal empathetic chatbot, named empathyear speech encoder vision encoder text embedding llm speech generator text response i understand how you feel. fear of heights can be very overwhelming. i understand how you feel. fear of heights can be very overwhelming. i am extremely afraid of going near heights. yeah, last year i got on the elevator game and i almost cried hahaha. yeah, last year i got on the elevator game and i almost cried hahaha. i understand how you feel. fear of heights can be very overwhelming. i understand how you feel. fear of heights can be very overwhelming. i am extremely afraid of going near heights. yeah, last year i got on the elevator game and i almost cried hahaha. yeah, last year i got on the elevator game and i almost cried hahaha. it's okay to feel scared. if you ever need someone to talk to about you fear ... it's okay to feel scared. if you ever need someone to talk to about you fear ... encoding reasoning generting meta-response emotion response content scene context agent profile talking-face generator are integrated into the llm's frontend to interpret different input modalities. the llm employs chained reasoning to sequentially infer and output a meta-response, encompassing emotion, scene context, response content, and agent profile. this holistic understanding and planning ensure the consistency of the text, sound, and visual outputs in terms of content and emotion, enhancing predictability and interoperability. further, the overall system is trained through a series of emotion-aware instruction-tuning to ensure comprehensive emotional understanding and generation capabilities. overall, this work pioneers a dialogue system that supports avatar-based multimodal empathetic response generation, marking an advancement toward emotional intelligence: 1) empathyear excels in accurately understanding user queries and generating high-quality responses across text, speech, and visual modalities with semantic and emotional coherence. 2) empathyear precisely perceives emotional semantics, supporting 32 types of emotions for both explicit and implicit types. 3) empathyear covers over 200 realistic scenarios, flexibly creating diverse digital avatar profiles. 4) while generating multimodal responses, empathyear also provides detailed rationales for decision-making, significantly enhancing interpretability. in efforts to construct empathetic dialogue systems, prior research @xcite @xcite has relied on detecting emotional signals within the given context, followed by generating responses that maintain emotional congruence. furthermore, some studies @xcite have incorporated external commonsense knowledge to achieve a deeper understanding of emotions and to facilitate empathetic responses. recently, there has been an explosion in llms (openai, 2022b,a; chung et al., 2022), demonstrating robust capabilities for content comprehension and reasoning. these advancements have facilitated superior erg performance @xcite . however, as mentioned earlier, current research in erg lacks a multimodal perspective, limiting its practical application value. this work also pertains to multimodal llms (mllm), wherein backbone llms serve as the pivotal centers for semantic and emotional reasoning and decision-making @xcite . the community has seen the emergence of various mllms, such as llava @xcite , blip2 @xcite , and minigpt-4 @xcite , etc. yet, most mllms are confined to understanding input multimodal information while falling short in flexibly outputting content across various modalities, including audio and visual content beyond text, e.g., image and video @xcite . as far as we are aware, next-gpt @xcite has accomplished any-to-any modality understanding and generation across four common modalities. however, next-gpt is primarily constrained to general scene and signal comprehension, with notable limitations in emotion detection and the generation of emotional content, due to two principal factors: firstly, the next-gpt architecture, lacking a talking head generator and a speech generator, cannot produce a talking face avatar or fluent speech. this prevents next-gpt from achieving multimodal erg, which is the key objective of our work. more importantly, next-gpt has not undergone specialized emotionaware fine-tuning, thus its ability to capture contextual emotions-particularly those that are implicit-is compromised. to overcome these limitations, our system has contemplated a series of emotion-reinforced learning techniques, for enabling stronger emotion perceiving. ## system workflow here we elaborate on the system's workflow from a high-level perspective. we conceptualize the system and the user as two entities, where empathyear processes the user's query and returns a response, while the user, in turn, provides a new query. from receiving a user's input request to generating a complete multimodal empathetic response, empathyear takes multiple sequential steps. figure 2 depicts the system's workflow. ▶ step-1. passing user query/api calling. our system will accept user input through a website interface or via predefined apis. it supports text inputs, voice (speech) inputs, or video input where the user is talking. ▶ step-2. encoding user&context (multimodal) inputs. the content input by the user, along with the historical dialogue context, is encoded. if the user's input is solely text, it is directly fed into the llm; if it includes multimodal information, it is first passed through a multimodal encoder before being input into the llm. ▶ step-3. generating meta-response with llm. the llm fully comprehends the input content, making corresponding decisions: outputting a meta-response that encompasses the understanding of emotion, scene understanding, the text response to be returned to the user, and the positioning of the agent profile. this compo-nent will be elaborated in section 4.2. ## implementation specification this section gives the specific implementation of empathyear, including the architecture, multimodal content generation, and learning methods. ## performance and quantitative analysis we finally quantitatively assess the exact performance of the system. ## conclusion we introduce empathyear, a novel, open-source, avatar-based multimodal empathetic chatbot. by employing an llm at its core, enhanced with multimodal encoders and generators, empathyear supports user inputs from any of text, sound, and vision modalities, and more importantly, producing multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized voices. empathyear allows for a richer, more empathetic communication experience, surpassing the limitations of current text-only erg systems, thus offering emotionally resonant interactions across a broader spectrum of scenarios. the system sets a new standard for human-level empathetic dialogue systems, blending intelligence with the ability to understand and express human emotions.