Modeling Empathetic Alignment in Conversation (2024)

Abstract: Empathy requires perspective-taking: empathetic responses require a person to reason about what another has experienced and communicate that understanding in language. However, most NLP approaches to empathy do not explicitly model this alignment process. Here, we introduce a new approach to recognizing alignment in empathetic speech, grounded in Appraisal Theory. We introduce a new dataset of over 9.2K span-level annotations of different types of appraisals of a person’s experience and over 3K empathetic alignments between a speaker’s and observer’s speech. Through computational experiments, we show that these appraisals and alignments can be accurately recognized. In experiments in over 9.2M Reddit conversations, we find that appraisals capture meaningful groupings of behavior but that most responses have minimal alignment. However, we find that mental health professionals engage with substantially more empathetic alignment.

## introduction empathy is a key aspect of successful clinical health conversations @xcite . in general, empathy involves an emotional component, where a listener resonates with the emotional tone of a speaker, and a cognitive component, conveying the listener understands the speaker @xcite . underlying both of these components is the perspective-taking by the listener to mirror the experience of the speaker, or as @xcite describes it, "being aligned is another way of being empathic." while past computational work on empathy has measured how empathetic messages can be, we still understand little about what aligns the language and perspective. here, we examine empathy as an alignment task, studying therapeutic conversations on reddit. given the importance of empathy, particularly in the clinical setting, nlp methods have attempted to model the relative level of empathy in replies @xcite . better models for recognizing empathy are aimed to help support generating more empathetic responses (e.g., @xcite . however, as @xcite note, many of these works focus only on the emotional mirroring component of empathy, rather than its cognitive component of perspective taking, and none explicitly model the alignment between the speaker, known as the target, and listener, as the observer. here, we introduce a new dataset and computational models for studying empathetic alignment in conversation. to quantify alignment, our work draws on the appraisal theory @xcite , which describes six aspects of how a person may experience a situation, e.g., describing its pleasantness or how much control they had, and encompasses both cognitive and emotional components. this scheme gives us a finegrain labeling of both what is described and how the person feels. because both the target and observer can appraise the same content differently, this view provides critical insight for understanding whether the two are aligned. this paper offers the following four contributions to the study of empathy in nlp. first, we introduce aloe, a new dataset, of therapeutic reddit conversations labeled with 9,284 appraisals from both the target and observer and 3,262 alignments between the target and observer. our dataset goes beyond theory to introduce new categories that model common types of aligned spans. second, in experiments, we show that appraisals can be accurately recognized and that the alignment between appraisals can be recognized, though we show that both are challenging tasks. third, in analyses on the appraisals and alignments of 2.3m posts and 8.9m comments, we show that appraisals meaningfully capture differences in how individuals experience distressing situations and in how others reply-but that the dominant form of align-ment is to reply with advice, rather than a matched appraisal. fourth, in comparisons between mental health professionals and laypeople on reddit, professionals have much higher alignment with targets; but, as seen in clinical settings, both professionals and laypeople decrease in their levels of alignment as they become more experienced. ## empathy in therapeutic settings empathy has been an important concept in social, personality, and clinical psychology @xcite @xcite . though being diversely defined, the most discussed aspects are emotional empathy and cognitive empathy @xcite . emotional empathy focuses on the vicarious sharing of emotion, while cognitive empathy relates to mental perspective-taking @xcite @xcite . in other words, emotional empathy is expressed as "i feel what you feel", and cognitive empathy is more commonly recognized as "i understand what you feel" @xcite . empathetic conversation is thought to play an important role in the development of social relationships @xcite , and mental health professionals are taught to develop empathetic skills @xcite , to improve patient outcomes and experiences. central to these empathetic conversations is the alignment between what a target is feeling and confirmation that the observer's mental model of the target matches these feelings; explicit expressions of this alignment are important for a target to experience an observer's response as empathetic (e.g., @xcite @xcite . while related to concepts like "active listening" or "reflective listening," this type of speech requires a communication of the observer's theory of mind to show that they have understood what the target has experienced, rather than just repeating parts of what a target has said. individuals seeking mental health support increasingly turn to social media @xcite . compared with traditional therapy sessions, the observers are no longer guaranteed to be trained professionals and the interactions are largely text-only. given abundant data and unique features, empathy in online communities becomes a valuable subject for active research @xcite , including comparisons of defining and expressing empathy between laypeople and professionals @xcite . within nlp, significant work has been done in predicting empathy @xcite , analyzing empathetic expressions and behaviors @xcite , and facilitating empathetic conversations @xcite @xcite . however, issues have been pointed out where empathy definitions are absent or abstract, and emotional empathy is overemphasized, while cognitive empathy is often absent or minimized @xcite . nlp models for recognizing empathy typically treat empathy as a classification or regression task. however, this introduces a gap: in clinical settings, speaking with empathy is often viewed as aligning the observer's speech to the target's, yet we lack methods for how to explicitly identify this alignment. our work directly addresses this gap by recognizing cognitive and emotive appraisals @xcite @xcite and measuring empathy in terms of the degree of observer alignment with a target's situation and appraises it in the same way. ## a dataset of empathetic appraisals to facilitate research on cognitive and emotional empathy, we introduce a new dataset of target and observer pairs, aloe (aligment of empathy), annotated for how each appraised the target's situation and which appraised passages are aligned. ## classifying appraisals and alignment models were trained to identify spans of appraisals and to align spans between target and observer. ## appraisal behavior different types of distressing events may be more likely to evoke specific appraisals, such as (un)pleasantness for the loss of a loved one, or the effort involved to handle mental illness. the 91 communities in our data cover a range of possible situations and @xcite note that empathy must be understood in context, with responses that adapt to the circumstances. here, we test whether individuals in these communities show regularity in how they appraise as targets and, do observers, in turn, vary the appraisals with which they respond. setup pca is then run on a matrix of subreddits and their normalized distribution of appraisals across all their posts. results communities were thematically clustered solely based on the relative distribution of appraisals (not content), shown in figure 1 and figure 2 . for example, for targets, clusters are seen for communities focused on structured therapy and self-help modalities (top left), recipients of abuse (bottom left), and various topics of grief (center to center right). similar clusters are also seen based on how observers appraise in these subreddits. although these subreddits all contain distressing situations, there is no a priori reason to expect that they should differ in how people categorize their lived experiences-many distressing situations could easily be described with any of the appraisals, yet individuals show regularity in how they rationalize and describe their thematicallysimilar experiences. this emergent grouping suggests that the related situations that targets find themselves in between these communities, while different, lead to similar ways of appraising those situations. the behavioral differences in how observers appraise from our large-scale observational results support the lab study of @xcite who found that observers vary the themes of their responses based on the type of suffering described by the target. ## do observers align? targets experience responses as highly empathetic when an observer appraises the situation in the same way @xcite , which requires that their appraisals align with those of the target. given the behavioral similarities seen between targets and observers in which appraisals they use, here we test whether the responses align. setup we calculate the probability that an observer o's appraisal of type a j is aligned the each type a j when used by the target t : p(a o i |a t j ). results in aggregate, observers only partially aligned with how the targets appraised (experienced) their situation (figure 3 ). instead of having the same appraisal, the majority of the observer's aligned text was giving advice to the target about a particular aspect. giving advice is a well-known aspect of reddit support communities (e.g., de choudhury and de, 2014) and some individuals so seek out communities for such advice (e.g., @xcite . while advice is not considered a component of empathy-and in some circumstances is considered counter-productive when used in empathetic situations like counseling @xcite )-its frequency does highlight its importance in the lived experience of support-seeking individuals. indeed, @xcite note that the experience of empathy in everyday situations encompasses a much broader set of behaviors than those listed in academic definitions. nevertheless, we do see a strong diagonal trend in figure 3 that suggests that, when not giving advice, observers do frequently align in their appraisals, suggesting empathetic behavior. two offdiagonal trends also emerge. first, observers frequently respond with certainty; in our annotation, we found that these were frequently gestures meant to reassure the target of their choice or action, and, thus, may be viewed as a type of compassionate response. second, observers often respond to a target describing the objective experience with comments about the target's agency (or not) in the situation; here too we find a type of compassionbased response where observers use agency language to deflect responsibility from the target onto other parties mentioned in the target's experience. appendix f reports additional details on specific subreddits' differences and behaviors. ## alignment by professional observers subreddit communities contain a mixture of observers, some of whom have professional training in mental health or medical domains. such training frequently includes discussions of empathetic and patient-centered dialog @xcite . some communities allow users to include a flair next to their username to indicate a selfreported qualification, such as a phd in psychiatry. given that users with such flairs should have experienced some training on how to behave more empathically-i.e., more alignment-here, we test the level of alignment of different professions of observers relative to the general public in our data. setup we used the flair on reddit as an indicator of whether a user is a professional or not. we adopt the flair-profession categorization of @xcite to map the text of each flair to a specific profession: 1 counselor, funeral role, medical doctor, nurse, psychiatrist, psychologist, psychotherapist, social worker, or therapist; see appendix e for details. flairs not indicating a specific degree or known license were left unmapped. for those with professional degrees, we also extract any reported status in training as either fully licensed or a student. professionals are defined as those who are licensed and have a non-student title, while laypeople are authors who do not have any student or pro1 we note that some professions overlap in their theme. for example, psychiatrists, psychologists, psychotherapists, and social workers may all be considered therapists or counselors. however, some qualifications, such as a licensed professional counselor (lpc) do have an associated degree. fessional flair throughout their usage of reddit. we further filtered the replies from the professionals when they were of their highest/most recent training level. ultimately, we identified 14,648 users as professionals, 2978 as students, and 1,686,362 laypeople in our data for analysis. appendix alignment is measured as the percentage of appraisals in the target's message that have an alignment in the observer's. we report the mean percentage for laypeople and each profession's users, calculated over all data in our dataset. results mental health professionals have higher alignment than laypeople, shown in figure 4 , both for aligning with the same appraisals and in general. a small split can be seen within professions as well: professions for clinical therapy (therapist, social worker, counselor, and psychologist) have among the highest alignment with targets, while medical professions (psychiatrists and doctors) are much lower-even lower than laypeople at matching the same alignment. our results suggest that the training received by mental health professionals does lead to higher alignment than laypeople. does the flair itself drive behavior? individuals who list their professional degrees as flair in a subreddit often interact with others in different subreddits where no such flair is visible. with no explicit mention of their profession, there is less reputational harm in responding with less effort which could lead to lower alignment. to test whether flair visibility drives alignment, we fit a linear regression on the observer's percent of alignment with categorical factors for (i) the profession, (ii) the subreddit, and (iii) profession flair visibility. flair visibility leads to a small but significant increase in alignment (β=0.027; @xmath0.01); full regression results are in appendix note that this effect is also not due to the change in community, as the subreddit regression factor controls for relative differences in alignment between communities. ## does experience influence alignment? professional health practitioner training emphasizes the importance of empathetic communication. however, multiple studies have noted that this training period marks a high point, and doctors and nurses become less empathetic over time @xcite . our first question is whether we observe a similar drop-off after therapist students transition to their fully licensed roles in our longitudinal data. laypersons too may benefit from explicit feedback on what comments are considered helpful, likely learning how to engage more empathetically over time. redditors are known to offer such feedback when the response matches their supportseeking goals @xcite . as a second related question, we test whether laypersons become more empathetic as they receive such feedback on which comments were most helpful. setup for the first question, we collect all authors who have flairs with licensed or student training levels. we then collected all of their engaged conversations as observers and split their comments into licensed and student periods based on when the flair text changes. comparison is made between conversations involving licensed observers and student observers. for the second question, we use data from r/advice, which assigns flairs of different levels to users based on how many times they have replied as an observer and another user has replied to express gratitude for their comment. we treat these flairs as proxies for experience in writing helpful replies. we collected the replies from different experience levels (flairs) and calculated the mean percentage of alignment for each level. results our results show that students are often more empathetic than their fully licensed counterparts (figure 6 ). of these, only the drop for therapist users is significant at @xmath1.1 using an independent t-test. our observations mirror results from @xcite and @xcite showing that nursing and medical students decreased their empathy levels with patients as they received more training. one likely driver of such drop-off is compassion fatigue, where high levels of empathy towards targets in a therapeutic setting can lead to a decreased ability to feel compassion for others @xcite . a similar trend is seen for users in r/advice as they gain more experience making helpful comments (figure 7 ), where users initially comment h e lp e r s u p e r h e lp e r e x p e r t a d v ic e g iv e r m a s t e r a d v ic e g iv e r p h e n o m e n a l a d v ic e g iv e r a d v ic e g u r u a d v ic e o r a c le e n li g h t e n e d a d v ic e s a g e a s s is t a n t e ld e r s a g e e ld e r s a g e 0.550 0.575 0.600 0.625 0.650 0.675 0.700 perc alignment figure 7: mean alignment by the amount of gratitude received at the time of comment, shown as flair from r/advice ordered from least to most thanked. with very high alignment and then slowly drop in alignment during their continued engagement until they reach a state-state level. while statusseeking is a known strong motivator for reddit users @xcite , we hypothesize that as users engage more frequently, some drop off in empathetic alignment may come from social media fatigue, which can decrease psychological well-being @xcite and thus lead to a decreased ability to empathize. we also hypothesize, that similar to professionals, these users also experience compassion fatigue from engaging with distressing comments. ## discussion individuals seek support in online communities, yet the type of support they receive varies, and, as we show, may not be well-aligned or, when aligned, may be advice rather than validation of their experience. the result that healthcare professionals are more likely to give advice raises new research questions about how online and offline therapeutic practices may differ. future studies could examine (i) the qualitative differences in the advice of laypersons vs professionals, (ii) whether targets view this advice as more valuable or empathetic, and (iii) if possible, the underlying motivations for professionals to give more advice, despite their training. the models developed in this paper can help surface such examples for study. the task of empathetic alignment can provide new opportunities for advancing nlp. for example, generating empathetic responses is effortful for many people @xcite , in part, because of the mental work. nlp models for recognizing this alignment could be used for assistive technologies that lower the cognitive load for responding with high empathy, such as highlighting passages that an observer might respond to and assessing whether their responses match what a target has actually said. such tools could potentially provide lower-effort entry points into the conversation to help people engage. ## conclusion empathy requires perspective taking on the part of an observer to align their cognitive and emotional experiences with another. this study goes beyond prior work in nlp on empathy to make these empathetic alignments explicit and to identify how observers mirror (or miss) the types of perspectives described by targets. by developing a new dataset, aloe, and models for appraisals and alignments in empathetic dialogues, our work enables studies of how and when observers empathize. in a large-scale study of reddit, we show that individuals seeking mental health support do receive empathetic replies-but that many aligned responses are giving advice, rather than acknowledging their perspective. however, we also show that mental health professionals on reddit show much higher alignment than the general public. our data and model can support future studies on how to help identify and correct misalignment when drafting responses or suggest opportunities for new alignments. all data, models, and annotation materials are available at https://github.com/jessicayjm/ modeling_empathy_alignment and the annotation tool is available in a stand-alone form at https://github. com/jessicayjm/span_alignment_annotation_tool 11 limitations our study examines conversations on a public social media platform, reddit. thus, our results may not be generalizable to other settings such as in-person settings where modalities other than texts might also be significant, or where individuals can speak in full confidence of anonymity. however, having the benefits of using longer texts, we were able to perform analysis on more complex appraisals compared with methods using shorter texts such as tweet data or text messages. secondly, we only analyzed reddit posts and the top-level comments and replies to those posts. while these post-comment pairs offer the cleanest signal of individuals looking for mental health support, our focus necessarily limits empathetic conversation that may be happening in replies to comments. we view this as an opportunity for future work in multi-turn dialog in reddit. meanwhile, analysis from the reddit data still carries its significance for longer conversation analysis given the impressive number of users being active on the platform. a key limitation in annotation comes from the inaccessibility to the mindsets of original targets and observers. our dataset reflects third-party perceptions of the state of mind of annotators-a challenging task given that we lack information on who the targets and observers are. however, under the design of the annotation process, our annotations likely mirrored the process that the observers go through when assessing potential targets. further, our adjudication process ensured that multiple potential interpretations were considered when any message was ambiguous so that the most likely could be chosen. future work should be encouraged to collect first-hand annotations directly from targets and observers, as this is currently a missing dataset for the community. we should also be aware of the inherent biases of lived experience from annotators. though crowdsourcing could provide more diversity, the annotation task itself is complex and not immediately amenable to crowdwork without extensive validation. we hope that our work can set a baseline for further generalization and to test how annotator identity can influence perceptions of empathy and alignment. the general trends in our experiments rely on classifiers that imperfectly learn how to identify appraisals and how targets and observers align. given the challenge of these classification tasks, our initial models attain only moderate performance which could potentially influence our downstream results. as a result, we have taken care to only describe trends in aggregate and to report confidence intervals and standard errors wherever possible. while moderate in performance, our approach mirrors work in other nlp tasks such as framing (e.g., @xcite @xcite @xcite where models operate on nuanced, often social, data to identify a moderate number of labels in order to derive largescale trends when applying these classifiers at scale. nonetheless, social information remains challenging to recognize even for large language models @xcite and our dataset provides an opportunity for future work to improve performance at recognizing empathetic alignment, which can open new doors for more fine-grained analyses of empathetic behavior. last, our results are drawn from a primarily western social media context and a westerneducated annotation pool. this cultural backdrop likely limits the generalizability of our results to other cultures. in our work, the annotators were aware of the cultural context of the paper's data. other work will be needed to understand how individuals from a variety of cultural backgrounds appraise distressing settings and how they effectively engage with empathy.