Empathy Identification Systems are not Accurately Accounting for Context (2023)

Abstract: Understanding empathy in text dialogue data is a difficult, yet critical, skill for effective human-machine interaction. In this work, we ask whether systems are making meaningful progress on this challenge. We consider a simple model that checks if an input utterance is similar to a small set of empathetic examples. Crucially, the model does not look at what the utterance is a response to, i.e., the dialogue context. This model performs comparably to other work on standard benchmarks and even outperforms state-of-the-art models for empathetic rationale extraction by 16.7 points on T-F1 and 4.3 on IOU-F1. This indicates that current systems rely on the surface form of the response, rather than whether it is suitable in context. To confirm this, we create examples with dialogue contexts that change the interpretation of the response and show that current systems continue to label utterances as empathetic. We discuss the implications of our findings, including improvements for empathetic benchmarks and how our model can be an informative baseline.

## introduction empathy is a fundamental phenomenon that allows us to better communicate and relate with others. studies show that empathy is significantly correlated with counseling treatment outcomes @xcite . computer systems could be improved by adding the ability to understand empathy. epitome @xcite took a step towards understanding empathy in language, introducing tasks such as empathy identification and empathetic rationale extraction. models built using epitome have been used to build or evaluate empathetic dialogue systems @xcite @xcite or study social effects of empathy @xcite . in this work, we explore whether current models are effectively considering dialogue context (shortened to context for the rest of this paper). we show that a simple model that does not consider context can achieve strong results, and that a model from prior work does not change its predictions when we make substantial changes to the context. together, these results indicate that models are more limited than previously thought. we introduce an adapted version of micromodels @xcite , a simple and explainable approach that combines a set of models, with each model identifying a specific linguistic phenomenon. this approach performs much better than the epit-ome's model on five metrics, comparably on five metrics, and much worse on two. critically, we achieve this without any use of context. we inspect our model's behavior and find that it can achieve accuracy scores that are as good as or better than the epitome model's for empathetic rationale extraction with as few as three seed/training utterances for our model. we also conduct an experiment to probe epit-ome's behavior. we take utterances from empathetic responses and randomly insert them as part of the response in another context. despite these insertions mainly being nonsensical and non-empathetic, prior models nearly always predict these responses as empathetic, demonstrating that the models rely on the surface form of the response rather than contextual understanding. the authors of epitome noted that empathy is contextual; a "reaction to an emotional stimulation" or a "deliberate process of understanding and interpreting the experiences" of others. foot_0 however, current systems do not effectively account for context: they may identify empathetic style, but they do not consider whether a response is indeed a reaction to feelings and experiences. future work micromodel 1 micromodel 2 (iii) feature vector (vi) classifier (i) training data label (ii) micromodels task-specific classifier (ebm) micromodel outputs mm 1 (q) … mm 2 (q) response utterance q figure 1: the micromodel framework training process. notably, our approach does not use any dialogue context, yet still performs strongly on current empathy detection benchmarks, raising the question of whether current systems and benchmarks are accounting for context. epitome data consists of pairs of dialogues from talklife and reddit, and two tasks: empathy prediction (emppred) and empathetic rationale extraction (empre). given a seeker's post s @xmath0, ..., s im and a response r @xmath1, ..., r in , each response r i is annotated with an empathy level (none, weak, or strong) in the context of s i across three communication mechanisms: emotional reactions, interpretations, and explorations. 2 empathetic rationales are spans of text that provide evidence of empathy. they are annotated at the token-level, e.g., the response "i feel you. are you okay?" is represented as [1, 1, 1, 0, 0, 0], [0, 0, 0, 1, 1, 1], and [0, 0, 0, 0, 0, 0] for emotional reactions, exploration, and interpretations, with one digit per token. the goal of emppred is to predict the correct level of empathy given (s i , r i ) across each communication mechanism. the goal of empre is to correctly extract the rationale spans. ## related work limited access to treatment for mental health, along with a rise in demand for scalable yet highquality interventions @xcite , has led to an abundance of conversational systems that provide mental health support @xcite @xcite . 910 a critical capability for these systems is to understand and interact with empathy, as studies show that empathy is significantly correlated with counseling treatment outcomes @xcite . broadly, recognizing empathy within dialogue has been studied under the following contexts: online platforms @xcite @xcite , formal counseling settings @xcite @xcite , or social media interactions @xcite @xcite . for instance, @xcite examined the interactions between practitioners and nonpractitioners that provide support on reddit. @xcite and @xcite analyzed the language of condolence and empathy in various social platforms. other settings for assessing empathy include reacations to news stories @xcite . another direction in the study of empathy within dialogue includes empathetic response generation @xcite @xcite . in order to assess the empathy level of their systems, researchers often use models such as epitome. @xcite @xcite . our work demonstrates the pitfalls that researchers should be aware of when taking such approach. other efforts include a taxonomy of empathetic responses @xcite , fine-tuned language models for empathy @xcite , as well as empathy-lexicons @xcite . ## conclusion in this paper, we assessed whether empathetic systems are correctly taking dialogue context into account. we demonstrated that a simple model with no contextual understanding can achieve results comparable to the epitome model and better than all baselines. we find that these results can be achieved by simply checking for semantic similarity to just three utterances. we also found that epitome model nearly always classifies a response as empathetic regardless of context, as long as it contains one of these three utterances. we conclude that current empathy recognition models do not effectively take contextual information into account. future work on benchmarks should consider including examples that require contextual under-standing to answer (s: "i got promoted!" r: "that's terrible, i'm sorry."). work on models should consider comparing with micromodels, a simple and practical baseline that serves as a reference point for performance without contextual understanding. these changes will better inform progress on models that meaningfully capture empathy. the code for our experiments is publicly available at https://github.com/ michigannlp/micromodels.