Can Machines Resonate with Humans? Evaluating the Emotional and Empathic Comprehension of LM s (2024)

Abstract: Empathy plays a pivotal role in fostering prosocial behavior, often triggered by the sharing of personal experiences through narratives. However, modeling empathy using NLP approaches remains challenging due to its deep interconnection with human interaction dynamics. Previous approaches, which involve fine-tuning language models (LMs) on human-annotated empathic datasets, have had limited success. In our pursuit of improving empathy understanding in LMs, we propose several strategies, including contrastive learning with masked LMs and supervised fine-tuning with large language models. While these methods show improvements over previous methods, the overall results remain unsatisfactory. To better understand this trend, we performed an analysis which reveals a low agreement among annotators. This lack of consensus hinders training and highlights the subjective nature of the task. We also explore the cultural impact on annotations. To study this, we meticulously collected story pairs in Urdu language and find that subjectivity in interpreting empathy among annotators appears to be independent of cultural background. Our systematic exploration of LMs’ understanding of empathy reveals substantial opportunities for further investigation in both task formulation and modeling.

## introduction with large language models (llms) demonstrating impressive capabilities in generating naturally sounding answers over a broad range of human inquiries, more individuals turn to seek solutions and emotional comfort by interacting with llmsupported chatbots @xcite . they express thoughts, feelings and share their experiences, expecting deep understanding and sympathetic responses from a chatbot that can resonate with them, as shown in figure 1 et al. @xcite . this requires llms to first fully understand the event, the emotions, and the empathy in the narratives, and then to respond appropriately @xcite . @xcite proposed the task of measuring empathic similarity which assesses the similarity between two narratives describing personal experiences across four aspects: event, emotion, moral, and empathy, using a numerical score ranging from 1 to 4 (see more in section 3.1). however, both fine-tuned lms (bert, roberta) and fewshot prompted llms achieved low correlation with human annotations. error analysis on @xcite shows that models can recognize dissimilar story pairs with scores in range of 1-2, but struggle to distinguish fine-grained differences between similar story pairs in range of 2.5-4 (see section 3). this suggests that the nuanced patterns and relationships in similar pairs are not captured by current methods @xcite . we hypothesize that the lms used in these methods were primarily trained for semantic understanding tasks, rather than for emotion and empathy. they may capture the similarity between events, but they struggle to understand the complex social and emotional signals in human narratives @xcite , this is further discussed in following sections. moreover, naïve llm prompts may not fully empower llms' reasoning ability to make correct judgements. to this end, we explore multiple strategies to improve the accuracy of empathic similarity, including contrastive learning of lms, llm reasoning and fine-tuning with and without chain-of-thought (cot) @xcite . we achieve around 5-10% improvement in the pearson and spearman correlations. however, accuracy appears to be capped and cannot be improved beyond a certain value. this motivated us to speculate that gold labels might be problematic due to the subjective nature of judging emotion and empathy similarity. to investigate this, we randomly sampled ten story pairs from empathicstories and asked eight annotators with different backgrounds to annotate the similarity in terms of event, emotion, moral and empathy (defined in section 3.1). overall, low human agreement for all aspects was observed, especially moral, followed by empathy, emotion; the highest agreement was for the event. interestingly, annotators from the same country/culture, particularly who are friends, had much higher agreement than others. we further collected a new urdu dataset to explore the cultural impact on empathy similarity labeling, which revealed similar findings. in sum, this work presents three contributions: sentence embedding enhancement: many previous methods improve semantic embeddings by adjusting loss functions in training @xcite . simcse leverages contrastive loss @xcite , and esimcse applies momentum contrast strategy to increase the number of negative pairs involved in the loss calculation, showing notable improvements across multiple semantic textual similarity benchmarks @xcite . @xcite improved the knowledge graph embeddings by integrating semantic awareness into traditional loss functions like hinge loss. huang et al. (2024) use a multi-task hybrid loss, incorporating both skeletal and semantic embedding into loss functions for micro-gesture classification. therefore, we investigate a variety of loss functions such as contrastiveloss, cosinesimilarity-loss, cosentloss and angleloss, based on lms including mpnet @xcite , roberta (base, large) @xcite , deberta (small, large) @xcite to improve sentence embedding in representing empathic features. llm reasoning and fine-tuning: naive llm prompts may fail to fully leverage llms' reasoning capabilities, leading to poor accuracy on empathicstories dataset @xcite . recent advancements in zero-and few-shot prompting techniques can boost performance such as chain-of-thought prompting (cot) @xcite , least-to-most prompting @xcite , and search-based approaches like tree-of-thought (tot) @xcite . we prompt llms by cot and also fine-tune llms with reasoning trajectories, encouraging llms to think over before making the final decision. ## dataset and error analysis in this section, we first introduce the dataset used in this work and then we perform an error analysis for some baseline methods. ## enhancing empathy representation in order to improve the model's ability to recognize the connections between narratives that are semantically diverse yet empathically similar, we apply constrative losses in lm fine-tuning, and incorporate reasoning in llm inference and fine-tuning. contrastive learning of lms: we use a contrastive loss to enhance representation by bringing the embeddings of similar examples closer, while pushing such of dissimilar samples apart in the hidden space @xcite . this involves first grouping them into positive and negative pairs, and then applying a contrastive loss to learn patterns. in this task, we use annotated similarity scores to determine positive and negative pairs, setting a threshold of 2.5 as the boundary following the binning approach in @xcite . we explored various contrastive losses, including the contrastiveloss @xcite , an-gleloss @xcite and cosentloss 3 . we also re-implemented the approach using cosine similarity loss as a baseline. we used masked language models (mlms) including mpnet @xcite , roberta (base, large) @xcite as text encoders leveraging their bidirectional encoding capabilities and computational efficiency. llm reasoning and fine-tuning: we explored the potential of llms for the task of empathic similarity in both inference and fine-tuning, with two prompting strategies: gold label-guided explanation: based on the pair of full stories with the ground truth similarity scores, we asked the llm to analyze the story pair from dimensions such as theme, emotional content, characters, narrative structures, and overall empathy. the generated analysis served as the reasoning content in supervised fine-tuning (sft). we explored both full parameter and peft @xcite fine-tuning. we used llama3-70binstruct to generate explanations by prompts in figure 4 of appendix b.1. ## experiments we experimented with discriminative lms and generative llms with zero-shot and fine-tuning. ## collective human opinions to analyze the subjectivity in empathic similarity labeling, we first collected eight annotations for each pair of english stories. we then explored the impact of language and culture on empathy labeling by collecting and annotating urdu story pairs. ## conclusion and future work we proposed a variety of strategies to enhance model performance on empathic similarity, including the use of contrastive losses on lms, llm reasoning and fine-tuning. our experiments demonstrated 5-10% improvements compared to baselines. however, our analysis revealed the subjective nature of empathic similarity between narratives. collective human annotations on both english and urdu story pairs illustrated the low human agreement in empathy labeling, highlighting the inherent challenges of this task and indicating an upper bound that a model can reach. the annotation of urdu stories further exposed the cultural impact on empathic labeling. we find that many factors impact the inner annotator agreement in empathic similarity labeling, including the native language of annotators, the similarity of their background, experience, the closeness between them, and training process. considering these factors that affect the correlations between annotators in a unified framework would be an interesting direction to explore in future work, especially the studies across multiple languages. furthermore, we will explore task reformulations to reduce the variability and subjectivity, and we will try more robust approaches to effectively model empathy in narratives.