Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction (2024)

Abstract: Large Language Model (LLM)-enhanced agents become increasingly prevalent in Human-AI communication, offering vast potential from entertainment to professional domains. However, current multi-modal dialogue systems overlook the acoustic information present in speech, which is crucial for understanding human communication nuances. This oversight can lead to misinterpretations of speakers’ intentions, resulting in inconsistent or even contradictory responses within dialogues. To bridge this gap, in this paper, we propose PerceptiveAgent, an empathetic multi-modal dialogue system designed to discern deeper or more subtle meanings beyond the literal interpretations of words through the integration of speech modality perception. Employing LLMs as a cognitive core, PerceptiveAgent perceives acoustic information from input speech and generates empathetic responses based on speaking styles described in natural language. Experimental results indicate that PerceptiveAgent excels in contextual understanding by accurately discerning the speakers’ true intentions in scenarios where the linguistic meaning is either contrary to or inconsistent with the speaker’s true feelings, producing more nuanced and expressive spoken dialogues. Code is publicly available at: https://github.com/Haoqiu-Yan/PerceptiveAgent.

## introduction artificial intelligence (ai) agents @xcite are entities designed to replicate human-like intelligence and functionalities, serving as the essential building blocks of ai systems. an ideal agent should be capable of perceiving its environment with sensors, making informed decisions, and then taking actions in response to users or scenarios. recently, large language models (llms) @xcite @xcite have exhibited remarkable capabilities in diverse tasks, offering opportunities for building general ai agents that engage in human-like interactions, such as virtual assistants and intelligent robots. however, current text-only dialogue systems @xcite fall short in bridging the gap between experimental and realistic scenarios, where humans perceive and understand the world through diverse multi-modal information. thus, the integration of acoustic information into dialogues has the potential to foster the development of more human-like agents, thereby enhancing the empathetic experience they offer. empathetic responses involve two essential aspects: cognitive and affective empathy @xcite @xcite , which reflect an understanding of the human-talker's thoughts and feelings respectively. specifically, cognitive empathy involves understanding the human-talker's thoughts, perspectives, and described events, enabling the agent to provide responses relevant to the dialogue topic @xcite . conversely, affective empathy entails responding based on observed emotional expressions in the dialogue history, contributing to the nat-1 15009 uralness of synthesized speech @xcite @xcite . while recent works @xcite @xcite ) leverage llm's strong capabilities of contextual understanding and content generation to synthesize empathetic speeches, there remains a discrepancy between cognitive and affective empathy. this arises because cognitive content is preassigned before affective speech is deduced from latent representations of multi-modal dialogue history. recently, advancements in multi-modal content perception and generation have been achieved by various methods @xcite @xcite , where audio is represented as either recognized text with an automatic speech recognition model or discrete features with a speech encoder. however, while linguistic information in speech is predominantly captured by both discrete acoustic units and textual representations, acoustic features tend to be disregarded. this oversight can lead to misinterpretations of the speaker's intentions, resulting in discrepant or even contradictory responses within the dialogue history. as illustrated in figure 1 , the left scenario fails to consider the perspective of the listener while the right one barely understands or empathizes with the speaker's feelings. in this paper, we propose perceptiveagent, an empathetic multi-modal dialogue system that can discern deeper or more subtle meanings beyond the literal interpretations of words, based on speaking styles described in natural language. specifically, perceptiveagent first comprehends the speaker's intentions accurately by a perceptive captioner model that captures acoustic features from each speech within dialogues. subsequently, an llm module acts as the cognitive core, producing the relevant response content with a caption describing how to articulate the response. a multi-speaker and multi-attribute synthesizer (msma-synthesizer) is then developed to synthesize nuanced and expressive speech. our contributions include the following: ## methods as a multimodal dialog system, perceptiveagent is capable of audio modality perception and empathetic speech generation, which is achieved through the incorporation of prosodic informa-tion expressed in natural language. to capture prosodic features from speech inputs, we propose a novel speech caption model that aligns audio features with the latent space of a pre-trained language model. to enhance empathy and diversity of the simulated speech communication, a multispeaker and multi-attribute vocoder is developed. this vocoder synthesizes speech by conditioning on both response contents and captions of speaking styles, resulting in more engaging and realistic dialogues. ## 15012 the architecture of the vocoder comprises a speaker embedder, an attribute embedder and a hifigan vocoder. the speaker embedder uses look-up tables to embed speaker identities, while a set of controllable attributes including speed, emotion, energy and pitch are embedded by the attribute embedder. to synthesize expressive speech, discrete units are initially embedded and up-sampled through a series of blocks consisting of transposed convolution and a residual block with dilated layers. prior to duration prediction, this up-sampled sequence is concatenated with the speed embedding. the speaker embedding and style embedding are subsequently concatenated to each frame in the up-sampled sequence, which is transformed to a mel-spectrogram by the hifigan generator. ## case study figure 3 presents two cases comparing the response quality between speech-gpt3.5 and per-ceptiveagent. it demonstrates that by explicitly incorporating acoustic information through captions, the llm can more accurately comprehend the speaker's intentions and generate more accurate and contextually appropriate responses. the first and second examples illustrate scenarios where the speaker's intention either contradicts or aligns with the linguistic contents, respectively. the first example in figure 3 (a) depicts an unplanned meeting conversation between two friends. analyzing solely from the textual contents, it is suggested that the speaker b is extremely excited and delighted about this conversation. however, a closer examination of the key words of "lower vocal" and "subbed energy" in speaker b's caption reveals an evasive attitude towards the situation. consequently, when confronted with speaker a's question, "were you here waiting for me?", it can be inferred that speaker b is not inclined to engage in extensive conversation. the absence of nuanced captions poses a challenge for speech-gpt3.5, leading to a misinterpretation and generating a response that implies a strong desire to continue the conversation. in contrast, percep-tiveagent provides a response in accordance with the underlying meaning. therefore, despite po-tential inconsistencies between linguistic contents and speaker intentions disrupting the accuracy of dialogue context understanding, perceptiveagent, with the aid of captions, can effectively capture the speaker's intent by correctly discerning the acoustic information of speech. in the second example in figure 3 (b) , the speaker a receives a paper from his mother and intends to share it with his friends. it can be inferred that he is highly excited at the moment, as evidenced by the key words "treble tone" and "energetically" in the caption. recognizing speaker a's excited mood, the response generated by percep-tiveagent mirrors the same enthusiasm and curiosity, aligning well with the ground truth. however, speech-gpt3.5 fails to perceive speaker a's excitement and merely raises the question in a bland manner. thus, in scenarios where the textual contents coincides with the speaker's intent, our model can also provide responses that correspond to the context of the conversation. ## conclusion in this paper, we propose perceptiveagent, an empathetic multi-modal dialogue system capable of accurately discerning the speaker's intentions through the integration of perceptive speech captions and to respond with nuanced and expressive spoken dialogues. specifically, perceptiveagent comprises three cascaded modules: a speech captioner for intention discernment, an llm for comprehension through sensory integration, and an msma-synthesizer for expressive speech synthesis. initially, the system employs a perceptive captioner model to capture acoustic features from each speech within dialogues. subsequently, an llm module serves as the cognitive core, generating relevant response content with a caption conditioned on the comprehension of the speaker's intentions. an msma-synthesizer is then developed to synthesize expressive speech. experimental results indicate perceptiveagent's strong ability in empathetic response generation, closely aligning with the dialogue context in terms of linguistic contents and exhibiting high expressiveness in acoustic information. additionally, a case study demonstrates perceptiveagent's capability to accurately identify the speaker's intentions in scenarios where the literal interpretations of words are either contrary to or inconsistent with the speaker's true feelings. input speaker a: are you being british?! caption: in a hushed voice, she speaks rapidly. speaking softly, she maintains a highpitched voice. speaker b: no. not anymore. caption: employing a lower vocal quality and customary speaking speed, his expression radiates subdued energy. speaker a: why are you shopping here? you don't live in this neighborhood. were you here waiting for me? caption: her subdued energy doesn't affect the high pitch of her voice at a regular speed.