E mpathic S tories++: A Multimodal Dataset for Empathy Towards Personal Experiences (2024)

Abstract: Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants’ homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals’ empathy toward others’ stories based on their personal experiences, evaluated in two contexts: participants’ own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.

## introduction empathy is a fundamental pillar of interpersonal human interactions ranging from prosocial behavior to enhancing human connection @xcite . modeling and understanding empathy is a complex task, due to its inherently interpersonal and experiential nature: empathy is tied to neurological synchronizations between representations of self and other @xcite , and is dependent on a person's past experiences @xcite . interest in empathy within ai communities has grown in recent years, as systems advance in context-awareness, naturalness, and fluency, although they typically fall short in social reasoning @xcite . few prior works present datasets that are sufficient to capture the richness of human empathy responses during personal experience sharing. these datasets are limited in the following ways: (1) they are not captured inthe-wild. existing multimodal empathy datasets are sourced from in-lab, online, or acted settings, which may differ greatly from empathy expressed in natural conversations. (2) they are not longitudinal, capturing only one-shot interaction settings, despite empathy being dependent on a combination of many past experiences. (3) previous datasets are not self-labeled. while empathy can be inferred by external cues, it is an inherently subjective process, requiring self-reported labels for user-centric or personalized modeling. in this work, we present the empathicsto-ries++ dataset, a multimodal dataset collected from a month-long deployment of social robots inthe-wild. in this work, participants shared personal stories with the robot, read stories that were empathically similar to their own experience, and then reflected on stories they empathized with @xcite . our interaction design allows researchers to explore empathy in the context of personal experience sharing and understand the influence of users' past experiences on their empathy towards others' experiences. we address gaps in previous empathy and emotion recognition datasets through the following attributes: (1) participant data is captured in their own homes with a social robot. previous works have shown that users are more comfortable disclosing sensitive information with ai partners than with people @xcite . participants in our study shared emotionally diverse and vulnerable stories from the comfort of their own homes. (2) participants interacted with the robot over the course of a month, allowing us to obtain longitudinal data. ## related work relevant prior works span two major areas: (1) social psychological theory on the relationship between prior experience and empathy and (2) datasets containing emotion or empathy ratings used for social-emotional reasoning tasks. ## data collection we deployed 46 in-home robots, powered by chat-gpt, foot_1 to converse with participants and record data. we recruited participants through mailing lists, and participants explicitly consented to data sharing. our protocol was approved by our institution's ethics review board. five participants withdrew from data collection for reasons not related to the study protocol. data collection took place over the course of a month, and participants were asked to complete between 6-12 conversation sessions with the robot (compensated $60 for 12 sessions). figure 2 shows the robot station in the participants' home and our data collection setup. the use of robots for data collection normalizes speaker-dependent characteristics that could add noise to the data from in-lab, human-human studies or acted scenarios @xcite . while one might hypothesize that the use of a robot would users less expressive, prior work shows that embodied social agents still elicit empathy behaviors similar to that of human-human interaction (spitale robot asks the participant to reflect on the following four areas: ways in which they related to the narrator, identifying the emotions of the narrator, regulating or comforting the narrator, and high-level takeaways from the story that the participant could apply to their own life. ## dataset statistics and properties the empathicstories++ dataset comprises video, audio, and text data from 269 sessions collected from 41 distinct participants, along with selfreported survey and interaction data. each video is a .avi file recorded at 15fps, whose cumulative length is 3,180 minutes (53 hours). the total number of utterances is 53,80, or about 20 per session (fixed for each interaction phase), totalling 337,147 words (1,258 per session). figure 3a shows the distribution of video lengths across sessions, ranging from 2 to 29 minutes (@xmath0, s.d. = 4.5 min). figure 3b depicts a similar distribution for spoken word counts. these ranged between 40 and 3418 words (@xmath1, s.d. = 531 words). participants felt varying levels of empathy towards with the stories they received, as the distribution (figure 3c ) of their empathy ratings on 1-5 scale shows (@xmath2.3, s.d. = 1.2). figure 4 depicts the demographic information of the participants. figures 4a-4d show the distributions of age, gender, trait empathy, and trait absorption. measurements of the big 5 personality traits are shown in the radar chart figure 4e . the change in levels of social connection, compassionate love for humanity, and wellness (see figure 3 ) across the month-long study are shown in figure 4f . participants shared vulnerable and meaningful stories across diverse topics (appendix a). our dataset is notable in that it (1) is captured in-the-wild, in participants' homes (2) contains longitudinal data, with trait and state surveys, and (3) is self-annotated, which is crucial for a subjective psychological process like empathy. ## conclusion this paper presents empathicstories++, the first in-the-wild, long-term, multimodal dataset on empathy towards personal experiences, which can be used to quantitatively evaluate empathy as it relates to one's past experiences. our dataset is self-annotated with empathy ratings and psychometric surveys. we present and benchmark a task on predicting user empathy from their interaction contexts. we observe that modality selection impacts model performance and is context-dependent. in the story share phase, where textual context was dominant, gpt-4 with text input performed the best in most metrics. however, in the reflection phase, where introspective verbal and nonverbal expressions are abundant, using v+t inputs showed 26.28% improvement in average for f 1 score, demonstrating their proficiency in extracting meaningful insights from multi-modal inputs. our work provides a valuable resource for future work in empathetic ai, quantitative exploration of cognitive insights, and empathy modeling. we publicly release our dataset to foster advancements in social-emotional ai. another limitation of our experimental results is that we only ablated contribution of modalities, but did not further interpret behavioral cues that might influence model performance. as such, these results are less interpretable due to lack of additional fine-grained annotations. future work can obtain fine-grained annotations of the video data for empathy-relevant behavioral cues such as arousal, valence, self disclosure, etc. our dataset is a valuable resource for furthering research in empathy modeling for ai systems. novel future directions to explore could include personalized modeling of empathy patterns, using the longitudinal data as well as understanding cognitive insights behind when empathy arises in personal story sharing. with an initial learning rate of 1e-4 with a scheduler steplr that decays the learning rate by 0.1 (γ) every 5 epochs (step_size). for the loss function, we use the mean squared error (mse). for the dataloader, we first conduct oversampling based on the empathy ratings due to its imbalance distribution as shown in figure 3c . next, we separate participants into train/valid/test sets in the ratio of 0.7/0.2/0.1 to ensure the model does not see the participants who were in the train sets. all models except for gpt-4 and empatheticstoriesbart were re-implemented to output multimodal representations that can be used to calculate similarities of story embeddings. we follow the default model parameters from the original implementations.