CAISA at WASSA 2022: Adapter-Tuning for Empathy Prediction (2022)

Abstract: We build a system that leverages adapters, a light weight and efficient method for leveraging large language models to perform the task Em- pathy and Distress prediction tasks for WASSA 2022. In our experiments, we find that stacking our empathy and distress adapters on a pre-trained emotion lassification adapter performs best compared to full fine-tuning approaches and emotion feature concatenation. We make our experimental code publicly available

## introduction empathy is an important interpersonal function in communication settings from conversations between friends and family, to educational, medical, or other goal-oriented dialogues. in natural language processing research, automatic empathy recognition and generation are explored for motivations such as improved experiences with opendomain dialogue agents @xcite @xcite , analyzing supportive interactions in online forums @xcite @xcite , and for the development of educational and evaluative tools for counselor training @xcite @xcite in addition to other educational domains @xcite . yet empathy prediction is a challenge for current language technologies due to resource availability and difficulty defining a gold standard for the complex phenomenon. the lack of proper resources for empathy modeling limits the ability of the nlp community to more widely explore it. many studies, for instance, are on sensitive data that cannot be made public. there are some datasets that are publicly available that are built on social media platforms, or through specific data collection tasks, however, these are few and far between, and each have limitations due to inherent challenges in the collection and annotation process. a general challenge with studying empathy is how to define the concept concretely enough to obtain consistent and relevant gold standard annotations, as there are many highly varied definitions in psychology research @xcite . furthermore, empathy datasets in nlp are almost always annotated by others rather than the person having an empathetic experience @xcite or the person on the receiving end. such annotations thus indicate particular aspects of language identified by a removed observer, rather than provide insight into the effect that particular empathetic experiences have on language. toward this issue, @xcite developed the empatheticreactions dataset, which contains empathic concern and personal distress ratings based on self-evaluations of individuals' own empathetic experiences at the time of writing the text. these reactions are short essays in which the author describes their feelings as they would to a friend after reading an article meant to evoke empathy. this data may then enable analysis into the way the empathetic experiences impact or relate to produced language. the empatheticreactions dataset is used for predicting empathy and distress in the wassa 2022 shared task, enabling a large group of people to research empathy prediction on a standard and public set of data. in this paper, we present our experiments for empathy and distress prediction as part of wassa 2022. we explore adapters for the task since it is more efficient than full fine-tuning, which so far has not been explored for empathy prediction. following work on domain transfer, we also build a system leveraging additional empathy data, as the amount of empathy data is still sparse. ## background the ability to recognize empathy in text is important for advancing language technologies from dialogue agents to computational social science tools. as such, there is a growing body of research on automatic empathy recognition. many studies concern highly sensitive and important scenarios such as counseling and medical dialogues @xcite or are crisis-related @xcite but such resources are protected and cannot be made public. however, there are a number of recently proposed empathy datasets available to the public, which are consolidated by means such as collecting and labeling social media @xcite , or through collection tasks @xcite . annotating empathy involves a host of challenges. most datasets are annotated by someone who did not take part in the writing or conversation, requiring them to interpret how the author felt, rather than acquiring this information from the authors directly. also, there are various definitions of empathy across fields. generally, nlp has considered emotional empathy, despite the prevalence of other components of empathy in psychology @xcite . there have been valuable efforts to build resources for empathy identification, each operating upon different perspectives of empathy. sharma et al. (2020)'s epitome dataset, contains support-seeker and responder post pairs from reddit and has multi-faceted empathy labels on the responder posts. the responder posts are annotated with the degree of three different aspects of empathy (interpretations, emotional reactions, and explorations), 0 for absent, 1 for weak, and 2 for strong. as this scheme contains distinct labels for both emotional and cognitive aspects of empathy, this dataset is a valuable resource for pursuing empathetic modeling beyond emotional aspects. zhou and jurgens (2020) introduced a dataset post-response pairs from reddit where the post contains an expression of distress and the response is a condolence. while the final dataset contained one empathy score, the annotation process was strictly guided by a multi-faceted definition of empathy, the appraisal theory @xcite . under this definition, the degree of empathy is how closely the responder's appraisal of another person's situation matches the person's appraisal of their own situation. rashkin et al. (2019)'s empatheticdia-logues dataset contains conversations grounded in one of 32 emotions. during data collection, participants were instructed to converse with each other. dialogues contain emotion labels but not empathy labels. welivita and pu (2020) further annotated empathetic intents in this dataset. buechel et al. (2018) built the empatheticre-actions dataset based on batson's empathic concern -personal distress scale @xcite . under this view, there are two aspects of empathetic reactions, the level a personal distress experienced by the reactor ("suffering with something") and the level of empathy ("feeling for someone") while maintaining self-other separation. here, empathy involves emotional feelings such as compassion, warmth, and tenderness, whereas distress involves those such as worry, alarm, and grief. these datasets may differ stylistically due to their different domains. having this diversity is valuable so that we can study how empathetic communication may vary across contexts. however, as the volume of data across these datasets is still limited, it is important to understand if they can be leveraged together despite their differences. ## task and dataset this paper describes our system submitted for track 1 of the wassa 2022 task which concerns empathy and distress prediction in @xcite 's dataset of empathic reactions to news stories. empathetic reactions are captured in essays written by people who were asked to read an article that involves a harmful situation a write a response. participants were asked to rate their empathy after reading an article before writing their response. these ratings were self-measured using batson's empathic concern -personal distress scale @xcite , which contains multiple items that were averaged in order to obtain the gold ratings for empathy and distress. the task of track 1 of wassa 2022 was to predict the numerical values for empathy and distress on a continuous scale for the essays. systems were evaluated by pearson's r correlation between the predictions and the actual values in a test set. wassa provided an extension of the dataset to include the original news articles, demographics (age, gender, ethnicity, income, education level) and personality information. the extension also included emotion labels obtained using pretrained emotion detection models. ## system description adapters offer a lightweight tuning strategy alternative to full fine-tuning @xcite . with adapter-tuning, new initialized layers are inserted at each layer of the original pretrained network, and the new weights are fine-tuned while the original network's weights remain fixed. adapters have been shown to effectively perform at near state-ofthe-art levels while drastically improving efficiency @xcite @xcite . as reported by the wassa 2021 task @xcite , the most robust systems for empathy and distress modeling involved fine-tuning of transformer models such as roberta @xcite and electra @xcite . in our experiments, we attempt an adapter tuning approach @xcite ) motivated by their efficiency, and compare to full fine-tuning. furthermore, we experiment with leveraging a different empathy dataset, epitome @xcite . this dataset contains support-seeker and responder posts on reddit (as described in § 2). ## results and discussion results from our submissions to the post-evaluation phase on the test dataset are presented in the emotionstack outperformed all other models on the test dataset on both empathy and distress while we only explored the emp track's tasks of empathy and distress prediction, the performance of the emotionstack inspired us to submit predictions for the emo track, predicting emotions. we used the same model, only changing the label set-up from predicting one value to predicting the six emotion categories-sadness, neutral, fear, anger, disgust, and surprise. this approach ranked highly with a macro f1-score of 0.604. a confusion matrix for our classifier is shown in figure 2 . the results of the adapter approach are exciting as it allievates the heaviness of full fine tuning. adapters make it easy to leverage knowledge from other tasks learned on other datasets. in particular, we observe positive effects from using the pretrained emotion adapter on these tasks, which likely provides important emotional information relevant to empathic concern and personal distress. however, we see no improvement from using the epitome data. similarly, recent work found separate empathy types were found to have different effects on toxicity reduction @xcite . in preliminary experiments, we fine-tuned on only one of these aspects at a time, as we were interested in whether they have distinct effects and whether one or a combination of them is particularly well suited for our tasks. further work is needed to definitively understand the effect of epitome and it's aspects on empathy and distress detection in the empatheticreactions. given the sparsity of public empathy data, it is imperative for future work to better understand how the existing datasets can complement each other. ## conclusion we presented our models for empathy and distress prediction on the empatheticreactions dataset for the wassa 2022 shared task. we found that a stacked adapter composition with the wassa task adapter stacked on a pre-trained emotion adapter (emotionstack) outperformed other methods. this approach mitigates the costs of full fine-tuning while achieving comparable results. furthermore, this method required no additional features beyond the empathetic reaction text. we further discussed challenges of researching empathy in natural language processing. in future work, we could explore incorporating the personal features provided for the shared task. we plan to further explore the use of different empathy datasets together for empathy prediction.