A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support (2020)

Abstract: Empathy is critical to successful mental health support. Empathy measurement has predominantly occurred in synchronous, face-to-face settings, and may not translate to asynchronous, text-based contexts. Because millions of people use text-based platforms for mental health support, understanding empathy in these contexts is crucial. In this work, we present a computational approach to understanding how empathy is expressed in online mental health platforms. We develop a novel unifying theoretically-grounded framework for characterizing the communication of empathy in text-based conversations. We collect and share a corpus of 10k (post, response) pairs annotated using this empathy framework with supporting evidence for annotations (rationales). We develop a multi-task RoBERTa-based bi-encoder model for identifying empathy in conversations and extracting rationales underlying its predictions. Experiments demonstrate that our approach can effectively identify empathic conversations. We further apply this model to analyze 235k mental health interactions and show that users do not self-learn empathy over time, revealing opportunities for empathy training and feedback.

## introduction approximately 20% of people worldwide are suffering from a mental health disorder @xcite . still, access to mental health care remains a global challenge with widespread shortages of workforce @xcite . facing limited in-person treatment options and other barriers like stigma @xcite , millions of people are turning to text-based peer support platforms such as talklife (talklife.co) to express emotions, share stigmatized experiences, and receive peer support @xcite . however, while peer supporters on these platforms are motivated and well-intentioned to help others seeking we differentiate between no communication, weak communication, and strong communication of these factors. our computational approach simultaneously identifies these mechanisms and the underlying rationale phrases (highlighted portions). all examples in this paper have been anonymized using best practices in privacy and security @xcite . in therapy, interacting empathically with seekers is fundamental to success @xcite . the lack of training or feedback to layperson peer supporters results in missed opportunities to offer empathic textual responses. nlp systems that understand conversational empathy could empower peer supporters with feedback and training. however, the current understanding of empathy is limited to traditional face-to-face, speech-based therapy @xcite due to lack of resources and methods for new asynchronous, text-based inter-actions @xcite . also, while previous nlp research has focused predominantly on empathy as reacting with emotions of warmth and compassion @xcite , a separate but key aspect of empathy is to communicate a cognitive understanding of others @xcite . in this work, we present a novel computational approach to understanding how empathy is expressed in text-based, asynchronous mental health conversations. we introduce epitome, 1 a conceptual framework for characterizing communication of empathy in conversations that synthesizes and adapts the most prominent empathy scales from speech-based, face-to-face contexts to text-based, asynchronous contexts ( §3). epitome consists of three communication mechanisms of empathy: emotional reactions, interpretations, and explorations (fig. 1 ). to facilitate computational modeling of empathy in text, we create a new corpus based on epitome. we collect annotations on a dataset of 10k (post, response) pairs from extensively-trained crowdworkers with high inter-rater reliability ( §4). 2 we develop a roberta-based bi-encoder model for identifying empathy communication mechanisms in conversations ( §5). our multi-task model simultaneously extracts the underlying supportive evidences, rationales @xcite , for its predictions (spans of input post; e.g., highlighted portions in fig. 1 ) which serve the dual role of (1) explaining the model's decisions, thus minimizing the risk of deploying harmful technologies in sensitive contexts, and (2) enabling rationale-augmented feedback for peer supporters. we show that our computational approach can effectively identify empathic conversations with underlying rationales (∼80% acc., ∼70% macro-f1) and outperforms popular nlp baselines with a 4-point gain in macro-f1 ( §6). we apply our model to a dataset of 235k supportive conversations on talklife and demonstrate that empathy is associated with positive feedback from seekers and the forming of relationships. importantly, our results suggest that most peer supporters do not self-learn empathy with time. this points to critical opportunities for training and feedback for peer supporters to increase the effectiveness of mental health support (miner et al., 2019; imel et al., 1 empathy in text-based, asynchronous mental health conversations 2 our dataset can be accessed from https://bit.ly/ 2rwy2gx . empathy is a complex multi-dimensional construct with two broad aspects related to emotion and cognition @xcite . the "emotion" aspect relates to the emotional stimulation in reaction to the experiences and feelings expressed by a user. the "cognition" aspect is a more deliberate process of understanding and interpreting the experiences and feelings of the user and communicating that understanding to them @xcite . here, we study expressed empathy in text-based mental health support -empathy expressed or communicated by peer supporters in their textual interactions with seekers (cf. @xcite ). 3 these scales, however, have been designed for in-person interactions and face-to-face therapy, often leveraging audio-visual signals like expressive voice. in contrast, in text-based support, empathy must be expressed using textual response alone. also, they are designed to operate on long, synchronous conversations and are unsuited for the shorter, asynchronous conversations of our context. in this work, we adapt these scales to text-based, asynchronous support. we develop a new comprehensive framework for text-based, asynchronous conversations ( ## framework of expressed empathy to understand empathy in text-based, asynchronous, peer-to-peer support conversations, we develop epitome, a new conceptual framework of expressed empathy (fig. 1 ). in close collaboration with clinical psychologists, we adapt and synthesize existing empathy definitions and scales to textbased, asynchronous context. epitome consists of three communication mechanisms providing a comprehensive outlook of empathy -emotional reactions, interpretations, and explorations. for each of these mechanisms, we differentiate between -(0) peers not expressing them at all (no communication), (1) peers expressing them to some weak degree (weak communication), (2) peers expressing them strongly (strong communication). here, we describe our framework in detail using the following seeker post as context for all example responses: i am about to have an anxiety attack. emotional reactions. expressing emotions such as warmth, compassion, and concern, experienced by peer supporter after reading seeker's post. expressing these emotions plays an important role in establishing empathic rapport and support @xcite . a weak communication of emotional reactions alludes to these emotions without the emotions being explicitly labeled (e.g., everything will be fine). on the other hand, strong communication specifies the experienced emotions (e.g., i feel really sad for you). interpretations. communicating an understanding of feelings and experiences inferred from the seeker's post. such a cognitive understanding in responses is helpful in increasing awareness of hidden feelings and experiences, and essential for developing alliance between the seeker and peer supporter @xcite . a weak communication of interpretations contains a mention of the understanding (e.g., i understand how you feel) while a strong communication specifies the inferred feeling or experience (e.g., this must be terrifying) or communicates understanding through descriptions of similar experiences (e.g., i also have anxiety attacks at times which makes me really terrified). explorations. improving understanding of the seeker by exploring the feelings and experiences not stated in the post. showing an active interest in what the seeker is experiencing and feeling and probing gently is another important aspect of empathy @xcite . a weak exploration is generic (e.g., what happened?) while a strong exploration is specific and labels the seeker's experiences and feelings which the peer supporter wants to explore (e.g., are you feeling alone right now?). consistent with existing scales, responses that only give advice (try talking to friends), only provide factual information (mindful meditation overcomes anxiety), or are offensive or abusive (shut the f**k up) foot_1 are not empathic and are characterized as no communication of empathy in our framework. ## data collection to facilitate computational methods for empathy, we collect data based on epitome. ## model with our collected dataset, we develop a computational approach for understanding empathy. ## results next, we analyze how effectively we can identify empathy with underlying rationales using our computational approach. ## model-based insights into mental health platforms we apply our model to study how empathy impacts online peer-to-peer support dynamics. to only focus on conversations related to significant mental health challenges and filter out common social media interactions (e.g., merry christmas), we carefully select 235k mental health related interactions on talklife using a seeker-reported indicator. 6 seeker post original response re-written response i cannot do anything without getting blamed today. this day is getting worse and worse. days end, tomorrow is a fresh start. i'm sorry that today sucks, but tomorrow is a fresh start. an hour ago i was happy an hour later i'm sad. am i getting mad now? try mindful meditation which can control anxiety that's something i've struggled with too, and it really pains me to hear that you're dealing with the same thing. have you considered trying meditation? i've found it to be very helpful. we investigate (1) the levels of empathy on the platform, its variation over time, and examine the relationship of empathy with (2) conversation outcomes, (3) relationship forming, and (4) gender. here, we operationalize relationship forming as seeker "following" the peer supporter after a conversation (within 24hrs). we find that seekers are 79% more likely to follow peer supporters after an empathic conversation (total score of 1+ vs. 0) than after a non-empathic one (fig. 3c ). ## conclusion we developed a new framework, dataset, and computational method for understanding expressed empathy in text-based, asynchronous conversations on mental health platforms. our computational approach effectively identifies empathy with underlying rationales. moreover, the identified components are found to be important to mental health platforms and helpful in improving peer-to-peer support through model-based feedback.