A Comparative Multidimensional Analysis of Empathetic Systems (2024)

Abstract: Recently, empathetic dialogue systems have received significant attention.While some researchers have noted limitations, e.g., that these systems tend to generate generic utterances, no study has systematically verified these issues. We survey 21 systems, asking what progress has been made on the task. We observe multiple limitations of current evaluation procedures. Most critically, studies tend to rely on a single non-reproducible empathy score, which inadequately reflects the multidimensional nature of empathy. To better understand the differences between systems, we comprehensively analyze each system with automated methods that are grounded in a variety of aspects of empathy. We find that recent systems lack three important aspects of empathy: specificity, reflection levels, and diversity. Based on our results, we discuss problematic behaviors that may have gone undetected in prior evaluations, and offer guidance for developing future systems.

## introduction empathetic dialogue systems have received significant attention in recent years, with new models that incorporate emotion, common sense, knowledge graphs or other signals into language models @xcite @xcite . meanwhile some researchers have noted that recent systems tend to generate generic, trite responses @xcite . however, these observations have not been verified in a systematic way. we survey 21 empathetic dialogue systems, using new analysis methods to see what progress has been made. quantitatively comparing these systems pose multiple challenges. automated metrics such as bleu @xcite , meteor @xcite , or rouge @xcite compare the lexical overlap between the generated text and a "ground-truth" discourse. however, in dialogue, there is an unbounded space of valid responses that differ from a ground-truth sample, and researchers have shown that these metrics have only a weak correlation with human judgement @xcite . to mitigate this issue, recent empathetic systems often rely on human evaluations. these crowdsourced evaluations typically measure empathy level, fluency, and relevance, with the latter two measuring overall conversational quality, with no relation to empathy. using only one measure of empathy is an overly simplified assessment of an empathetic dialogue system @xcite , failing to account for how empathy is a multidimensional construct @xcite with a wide range of definitions in terms of social, emotional, or cognitive dimensions @xcite . one issue with using a single score is that some systems might be effective in one aspect of empathy, while other systems might excel in others, but that variation would be hidden when considering a single overall empathy score. another issue is that a single score does not provide information about the nature of remaining errors, which would be valuable for guiding future work. given these limitations of prior evaluation procedures, we comprehensively study nine systems on multiple metrics that are each grounded in a multidimensional definition of empathy. namely, we survey recent papers that propose an empathetic dialogue system from nlp conferences in the last three years. of the 21 systems that we identify, we further analyze every system that has been trained on empatheticdialogue @xcite , the most prominent dataset used by researchers ( our study provides a reflection on the advances made by recent empathetic systems, and offers valuable takeaways for the development of future systems. empathy frameworks. empathy is a nuanced human experience and a complex multidimensional construct which is difficult to computationally assess. broadly speaking, empathy has two aspects: emotion and cognition @xcite . the emotional aspect relates to the emotional reaction or connection that is formed as a reaction to one's emotions or experiences, whereas the cognitive aspect relates to the reflective and interpretive process of understanding one's experiences. definitions from psychotherapists provide the foundations for computational researchers to assess the empathy level of their systems. for instance, epitome @xcite identifies empathy across three "communication mechanisms" (emotional reactions, interpretations, exploration). @xcite grounds their emotional support conversation framework on hill's helping skills theory @xcite , consisting of three stages of support (exploration, insight, action). however, by surveying recent empathetic systems, @xcite critically indicate that most systems lack a clear definition of empathy. our findings confirm that of @xcite , and go beyond their work by providing empirical studies. ## limitations of current evaluations recent empathetic dialogue systems follow a common procedure to evaluate their output, which includes automated metrics and human evaluations. single-turn multi-turn however, they have several shortcomings, which we discuss in this section. namely, we survey the evaluation procedure of 21 empathetic dialogue systems published over the last five years at leading nlp venues, as summarized in ## multidimensional evaluations of empathy prior work from psychology that studies effective ways of evaluating empathy in counseling settings has defined various aspects of empathy @xcite @xcite @xcite . in this work, we focus on four specific aspects from truax and carkhuff (1964), @xcite , and houck et al. ( 2012 ), which we discuss below. for each aspect, we provide our motivation for measuring the aspect, our methodology for measurements, and our results. ## lessons learned based on our evaluations, we formulate a set of takeaways, as well as concrete suggestions for evaluating future systems. single vs. multidimensional empathy. single dimensional evaluations of empathy are not reflective of system improvements. unlike a single empathy score, our analyses allow us to attribute system behavior to specific aspects of empathy. overall, our takeaways echo the conclusions of prior work @xcite ) that recent systems rely on an overly simplified, single dimensional definition of empathy, and highlight the shortcomings of current evaluation methods (section 3). ## conclusion in this work, we surveyed recent empathetic dialogue systems. we discussed the shortcomings of the evaluation of these systems -relying on a single empathy score fails to capture the multidimensional aspect of empathy. by deploying several automated metrics, each grounded in different aspects of empathy, we identified multiple areas in which current systems could be improved, and uncovered behaviors that have gone undetected with previous evaluations, such as the lack of specificity, reflection levels, and diversity. furthermore, we found that newer systems do not necessarily lead to improved performance under our metrics. we highlight the challenges of evaluating empathetic systems, and propose possible approaches to measure meaningful progress on the task.