Multi-dimensional Evaluation of Empathetic Dialogue Responses (2024)

Abstract: Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents—that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both expressed intents from the speaker’s perspective and perceived empathy from the listener’s perspective. We apply our analytical framework to examine internal customer-service dialogues. We find the two dimensions (expressed intent types and perceived empathy) are interconnected, while perceived empathy has high correlations with dialogue satisfaction levels.To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on FlanT5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method’s strong performance.

## introduction empathy as a complex socio-emotional phenomenon plays a crucial role in forming and maintaining social interactions @xcite . prior works in psychology and social science have highlighted the importance of empathy in human communications @xcite @xcite . as speech and text-based dialogues remain prevalent in human communication, many previous studies had developed computational approaches to analyze and measure empathy expressed in dialogues @xcite @xcite . prior efforts had proposed empathy frameworks grounded in psychology and social science theories. these frameworks were then adapted for specific applications or dataset domains. correspondingly, researchers proposed data-driven approaches to analyze and measure empathy. upon reviewing existing works, we found that they mostly adopted an individualistic conceptualization of empathy @xcite @xcite which positions empathy as a primarily individual mental process. these works mainly focused on modeling expressed empathy in dialogues. however, recent psychology research @xcite argued that empathy is more fundamentally relational: instead of a static personality trait, empathy is a dynamic process, aiming to serve interpersonal communicative purposes. @xcite emphasized empathy as a collaborative practice, involving participation by both parties and is shaped in social interactions. thus, it is critical to further examine perceived empathy in a collaborative, dynamic dialogue setting. in this study, we propose a novel conceptual framework to evaluate empathy in multi-round dialogues. specifically, this framework measures both expressed and perceived empathy intents. our proposed multi-dimensional evaluation framework includes four perceived empathy dimensions: (1) perceived engagement, (2) perceived understand, (3) perceived sympathy and (4) perceived helpfulness. to validate this framework's practicality, we annotated an internal dataset of 2,000 customer service dialogue sessions. and performed a fine-grained analysis. our findings show a strong correlation between perceived empathy and overall dialogue satisfaction ratings, affirming the framework's efficacy in capturing empathetic traits. the proposed empathy evaluation framework still requires subjective assessments from human experts, which are expensive and non-trivial to collect. to scale up evaluation without excessive reliance on annotated data, we investigate two model-based evaluation approaches: prompting llms and training language model-based classifiers. we show that prompting methods with popular llms such as gpt-4 and flan family models perform poorly on our datasets, indicating the challenging nature of measuring conversational empathy. in contrast, traditional supervised fine-tuning yields better accuracy with labeled data. notably, instruction-finetuned classifiers based on flan-t5 family models @xcite achieve the best performance. we also conduct a comprehensive ablation study of this method to provide insights into its strong performance. to summarize, our contributions are three-fold: definition of empathy in literature. empathy has been an influential concept in ethics and moral theory @xcite @xcite , social/developmental psychology @xcite and other domains. contributions from various domains result in different definitions, functions and proposed empathy components, posing a challenge to studying empathy @xcite . prior works @xcite @xcite , inter alia) mainly consider empathy as an individual's mental process. recent psychological studies highlight empathy as a collaborative practice involving participation by both parties @xcite @xcite . within this work, we adopt this interactional definition of empathy, as our aim is to measure and evaluate empathy in social scenarios such as real-time human-human conversation. ## empathy measurement models this section describes model-based metrics to measure conversational empathy. similar to previous studies @xcite @xcite , inter alia), we leverage language models to build classifiers to distinguish between empathetic and non-empathetic utterances in dialogues. measuring conversational empathy is an intrinsically challenging task because: (1) empathy is often communicated implicitly @xcite @xcite , posing challenges for state-of-the-art language models @xcite ; (2) empathy expression styles and intents are often domain specific; and ## conclusion and future work in this study, we introduce a comprehensive framework for assessing conversational empathy, focusing on both expressed empathetic intentions and perceived empathy. we apply this framework to analyze our internal dataset and find that it effectively correlates expressed intentions, perceived empathy, and overall dialogue satisfaction. this proposed conceptual evaluation framework requires subjective assessments from trained annotators, which can be expensive and non-trivial to collect. to explore automated empathy measurement models, we rigorously compare various methods against human judgments. our instruction finetuning technique achieves the highest classification performance, measured by f-score and accuracy compared to human-based judgments. while our discussion primarily centers on measuring empathy perception in human-human conversations, an important future direction is extending this analysis to human-machine interactions.