Modeling Empathic Similarity in Personal Narratives (2023)

Abstract: The most meaningful connections between people are often fostered through expression of shared vulnerability and emotional experiences in personal narratives. We introduce a new task of identifying similarity in personal stories based on empathic resonance, i.e., the extent to which two people empathize with each others’ experiences, as opposed to raw semantic or lexical similarity, as has predominantly been studied in NLP. Using insights from social psychology, we craft a framework that operationalizes empathic similarity in terms of three key features of stories: main events, emotional trajectories, and overall morals or takeaways. We create EmpathicStories, a dataset of 1,500 personal stories annotated with our empathic similarity features, and 2,000 pairs of stories annotated with empathic similarity scores. Using our dataset, we fine-tune a model to compute empathic similarity of story pairs, and show that this outperforms semantic similarity models on automated correlation and retrieval metrics. Through a user study with 150 participants, we also assess the effect our model has on retrieving stories that users empathize with, compared to naive semantic similarity-based retrieval, and find that participants empathized significantly more with stories retrieved by our model. Our work has strong implications for the use of empathy-aware models to foster human connection and empathy between people.

## introduction through personal experience sharing, humans are able to feel the sting of another person's pain and the warmth of another person's joy. this process of empathy is foundational in the ability to connect with others, develop emotional resilience, and take prosocial actions in the world @xcite @xcite . today, there is more visibility into the lives of others than ever before, yet loneliness and apathy are widespread @xcite @xcite . while these challenges cannot be solved with technology alone, ai systems can be developed to bolster emotional support, empathy, and truly meaningful connections through fostering personal experience sharing @xcite @xcite . in order to do so, these systems must be able to reason about complex social and emotional phenomena between people. in this work, we introduce the task of modeling empathic similarity, which we define as people's perceived similarity and resonance to others' experiences. for example, in figure 1 , empathic similarity aims to capture that narrator a, who feels lonely in their small town, is likely to empathize with narrator b, who is feeling isolated at their new job. crucially, empathic similarity differs from traditional notions of textual similarity that have been the main focus of nlp work (e.g., semantic similarity; @xcite ; narrator a will likely not empathize with narrator c, despite both stories having higher semantic similarity. we operationalize empathic similarity around alignment in three features of a personal story (highlighted in figure 1 ): its main event, its emotional reaction, and its overall moral or story takeaway @xcite @xcite @xcite @xcite , as motivated by social psychology and narratology literature. from our definition, empathic similarity arises from the interplay of the main events, emotions, and morals in story, where some components or all components must be similar in order for two narrators to resonate with one another. for example, narrator a and b both experience loneliness, even though their actual situations are different (living in a small town versus working at a company). to enable machines to model empathic similarity, we introduce empathicstories, foot_0 a corpus of 1,500 personal stories, with crowdsourced annontations of the free-text summaries of the main event, emotion, and moral of the stories, as well as an empathic similarity score between 2,000 pairs of stories. we find that finetuning on our paired stories dataset to predict empathic similarity improves performance on automatic metrics as compared to off-the-shelf semantic similarity methods. while automatic evaluation a valuable signal of model quality, it is crucial to showcase the realworld impact of our task on improving empathy towards people's stories. as such, we conducted a full user study with 150 participants who wrote their own personal journal entries and were presented stories retrieved by our model (and by a semantic similarity baseline). our results show that users empathize significantly more with stories retrieved by our finetuned empathic similarity model compared to those from a semantic similarity baseline (sbert; @xcite . our findings highlight the applicability of our framework, dataset, and model towards fostering meaningful human-human connections by enabling nlp systems to reason about complex interpersonal social-emotional phenomena. ## related work document similarity is a well-defined task in nlp @xcite @xcite , but few have applied this work to matching personal narratives based on shared emotional experiences @xcite . one study used latent dirichlet allocation (lda) to cluster cyberbullying stories and match these stories based on similarity in theme @xcite , but discovered that only 58.3% found the matched story to be helpful if provided to the narrator of the original story. other work has explored ways to bridge the features of a story and human-perceived similarity of stories @xcite . @xcite found that people use labov's action (series of events) and evaluation (narrator's needs and desires) clauses to identify similarity in personal narratives @xcite . their findings support our decision to focus on modeling events, emotions, and morals within stories. most relevant to our work are recent advances in social and emotional commonsense reasoning using using language models. specifically, prior methods have used finetuning of language models such as bert @xcite and gpt-2 @xcite to model events and the emotional reactions caused by everyday events @xcite @xcite @xcite @xcite as well as predicting empathy, condolence, or prosocial outcomes @xcite @xcite @xcite . understanding the emotional reactions elicited by events is a challenging task for many nlp systems, as it requires commonsense knowledge and extrapolation of meanings beyond the text alone. prior works use commonsense knowledge graphs to infer and automatically generate commonsense knowledge of emotional reactions and reasoning about social interactions @xcite @xcite . however, there are still many under-explored challenges in developing systems that have social intelligence and the ability to infer states between people @xcite . in contrast to previous works, we present a task for reasoning between pairs of stories, beyond predicting social commonsense features of texts alone. our work builds on top of prior work by developing a framework around empathic resonance in personal narratives in addition to assessing the human effect of ai-retrieved stories on empathic response beyond automatic metrics. unlike previous works, our human evaluation is a full user study to see how the model performs given a story that the users told themselves, which is much more aligned with real-world impact. ## empathic aspects of personal stories modeling empathic similarity of stories requires reasoning beyond their simple lexical similarities (see figure 1 ). in this section, we briefly discuss how social science scholars have conceptualized empathy ( §3.1) and draw on empathy definitions relevant for the nlp domain @xcite . then, we introduce our framework for modeling empathic similarity of stories and its three defining features ( §3.2). ## empathicstories dataset we introduce empathicstories, a corpus of personal stories containing 3,568 total annotations. specifically, the corpus includes empathic similarity annotations of 2,000 story pairs, and the main events, emotions, morals, and empathy reason annotations for 1,568 individual stories. an overview of our data annotation pipeline is shown in figure 2 and data preprocessing steps are included in appendix d. in appendix h, we show that using llms for human annotation is not viable for our task. ## modeling empathic similarity to enable the retrieval and analysis of empathically similar stories, we design a task detailed below. in appendix b, we also propose an auxiliary reasoning task to automatically extract event, emotion, and moral features from stories, which could be used in future work to quickly generate story annotations. ## automatic evaluation to evaluate the quality of empathic similarity predictions, we first compare the spearman's and pearson's correlations between the cosine similarity of the sentence embeddings and the gold empathic similarity labels. next, we bin scores into binary similar/dissimilar categories (> 2.5 @xmath0.5 respectively) compute the accuracy, precision, recall, and f1 scores. finally, we compute a series of retrieval-based metrics including precision at @xmath1, kendall's tau @xcite , and spearman's correlation @xcite for the ranking of the stories (how close the overall rankings are). while gpt-3 and chatgpt have high performance on the precision at k retrieval metric, in practice, it is not feasible to prompt the models with every pair of stories in the retrieval corpus. ## user study prior work's versions of human evaluations @xcite @xcite are humans verifying or ranking model outputs based on inputs from test data. this provides a valuable signal of model quality, but isn't representative of how a model could be used in real-world applications due to input distribution mismatch and lack of personal investment in the task. our human evaluation is a full user study to see how the model performs in retrieving a story that is empathically similar to a story that the users told themselves. through our user study, we demonstrate the applicability of the task to improve empathy towards retrieval of human stories, as well as how our dataset was used to develop the empathic similarity retrieval task and why the task matters in the real-world. our hypothesis is: users will empathize more with stories retrieved by our model (bart finetuned on empathicstories) than stories retrieved by sbert. ## future directions for empathic similarity in summary, few prior works on text-based empathy have looked at modeling empathy in two-way interpersonal settings for human-to-human connection, as most focus on detecting empathy or generating empathetic utterances, and even fewer of these works have shown tangible outcomes in human studies. with increasing polarization, loneliness, and apathy @xcite , personal experiences are a fundamental way people connect, yet existing social recommendation is not targeted for human-human connectivity and empathy. empathically encoded story embeddings could be useful for a variety of nlp tasks, including retrieval, text generation, dialogue, and translation, for example in the following settings: we encourage future works to explore these directions in developing more human-centered approaches for interactions with nlp systems. ## conclusion this work explores how we can model empathic resonance between people's personal experiences. we focused specifically on unpacking empathy in text-based narratives through our framework of the events, emotions, and moral takeaways from personal narratives. we collected empathicstories, a diverse dataset of high-quality personal narratives with rich annotations on individual story features and empathic resonance between pairs of stories. we presented a novel task for retrieval of empathically similar stories and showed that large-language models finetuned on our dataset can achieve considerable performance gains in our task. finally, we validated the real-world efficacy of our bartfinetuned retrieval model in a user study, demonstrating significant improvements in feelings of empathy towards stories retrieved by our model compared to off-the-shelf semantic similarity retrieval. empathy is a complex and multi-dimensional phenomenon, intertwined with affective and cognitive states, and it is foundational in our ability to form social relationships and develop meaningful connections. in a world where loneliness and apathy are increasingly present despite the numerous ways we are now able to interact with technologybased media, understanding empathy, developing empathic reasoning in ai agents, and building new interactions to foster empathy are imperative challenges. our work lays the groundwork towards this broader vision and demonstrates that ai systems that can reason about complex interpersonal dynamics have the potential to improve empathy and connection between people in the real-world.