The uncivil empathy: Investigating the relation between empathy and toxicity in online mental health support forums (2023)

Abstract: We explore the relationship between empathy and toxicity in the context of online mental health forums. Despite the common assumption of a negative correlation between these concepts, it has not been empirically examined. We augment the EPITOME mental health empathy dataset with toxicity labels using two widely employed toxic/harmful content detection APIs: Perspective API and OpenAI moderation API. We find a notable presence of toxic/harmful content (17.77%) within empathetic responses, and only a very weak negative correlation between the two variables. Qualitative analysis revealed contributions labeled as empathetic often contain harmful content such as promotion of suicidal ideas. Our results highlight the need for reevaluating empathy independently from toxicity in future research and encourage a reconsideration of empathy’s role in natural language generation and evaluation.

## introduction natural language processing (nlp) technology has been instrumental in both the analysis and enhancement of online discussions, as exemplified by its application in platforms like reddit @xcite . specifically, the detection of toxicity in online comments has emerged as a widely embraced preventive measure for moderating online discussions @xcite . on the other hand, in recent years, there has been a surge of research interest in nlp on empathy @xcite , due to its critical role in human communication and relationship building @xcite . in the realm of online public discourse analysis, both toxicity and empathy are frequently studied and discussed within the broader context of civility @xcite . while toxicity is typi-cally characterized as a form of uncivil behavior, empathy is associated with civil interactions that contribute to pro-social outcomes. some research in the field of nlp has made unexamined implicit assumptions based on this conceptual contrast. one such assumption posits a negative correlation between empathy and toxicity @xcite . on the other hand, studies from psychology hold mixed views regarding the relation between the two concepts @xcite . while the effect and roles of both toxicity and empathy are complex, developing technology founded on unexamined assumptions entails the risk of unforeseen consequences. this study analyses the correlation between toxicity and empathy using the human annotated empathy labels of epitome, a widely used mental health subreddit empathetic dataset @xcite , and augmenting it with toxicity labels predicted by two popular apis. we conduct a qualitative analysis of epitome responses which are both empathetic and predicted as toxic. our key findings and contributions are: ## related work toxicity is generally defined as language that is harmful, offensive, or suppressing the expression of others @xcite . while earlier tox-icity detection tasks primarily focused on binary classification @xcite , more recent studies have shifted towards incorporating more specific fine-grained labels, such as personal attacks @xcite , hate speech @xcite and many more @xcite . recent developments also encompass toxic span detection @xcite and implicit, context-dependent toxicity detection @xcite . some of these advancements have transitioned into production as public apis, such as the perspective api @xcite , and find practical use not only in everyday applications like online forum moderation but also in research fields beyond computer science, such as political science. however, some concerns have been raised regarding the potential inconsistency and oversimplification in the underlying definitions of toxicity within the detection models @xcite . driven by the interest in developing more engaging and supportive ai agents, empathy has emerged as a prominent theme in recent nlp research @xcite . earlier research on empathy primarily focused on emotional understanding and reactions, whereas recent works delve into the cognitive dimensions of empathy, including perspective-taking @xcite . while numerous studies aim to generate empathetic responses resembling human ones, few concentrate on automated empathy detection. this trend can be attributed, in part, to empathy's diverse definitions, spanning various fields such as cognitive neuroscience and psychology @xcite . the epitome dataset @xcite stands out as the sole dataset to not only label empathy levels but also annotate empathy across three distinct components: emotion reaction (er), interpretation (ip), and exploration (ex), encompassing both emotional and cognitive aspects of empathy. ## methodology in this study, we use the sub-reddit version of the epitome dataset, which was sourced from 55 mental health focused subreddits. the dataset includes 3081 pairs of support seeker post and peer support response. each response message is human annotated with the levels (none: 0, weak: 1, strong: 2) of the three empathetic components (er, ip, ex). appendix a covers the detailed definitions and annotation level criteria. we use two widely-used apis for harmful and toxic online content detection, the perspective api (jigsaw, 2023) and openai's moderation api (ope-nai, 2023). perspective api is provided by google for online content moderation. the underlying models of the api are trained on online comment labels from a variety of sources, like wikipedia. given an input message, the api returns continuous scores (0-1) for 6 different toxicity categories. besides the score, the api also returns the detected toxic spans for each corresponding category. openai's moderation api was developed primarily for moderating the input and output of their flagship large language model chatgpt. with less emphasis on toxicity per se, the api is designed to detect harmful and dangerous content. for each input message, it returns an overall binary flag (0,1) and 11 continuous category scores (0-1). appendix b contains the detailed definitions for the labels of both apis. using both apis, we (automatically) annotate the peer support responses in the epitome datasets with toxicity labels. we are primarily interested in empathy/toxicity in epitome peer responses, and so feed only the responses into the apis. to clarify, we do not include the support seeker post as part of the input, and so the classification is done using only the response. both quantitative and qualitative analysis have been conducted based on the scores from the apis along with the human annotated epitome labels. to validate the predictions of the two apis and ensure the validity of this study, we conducted manual annotation on 50 positive (labeled as toxic by at least one api) and 50 negative samples (not labeled as toxic by either api), resulting 0.87 accuracy (details in appendix c), suggesting that the toxicity predictions are generally reliable. to provide a qualitative understanding on these predictions, we conducted an error analysis and identified that the predominant error cases (12 out of 13) were false positive errors. these errors were largely attributed to the predicted self-harm labels from the openai's moderation api (10 out of the 12), while a smaller subset were related to profanity use (2 out of 12). upon further qualitative analysis, we identified that the error cases frequently featured lengthy content with mixed intentions. for instance, these cases often began with individuals sharing their own suicidal thoughts or experiences but subsequently shifted towards discouraging suicide. we also observed that the apis can at times exhibit oversensitivity to the presence of specific keywords like "suicide", "depression" and "shit", even when these terms are used with the goal of emphasis rather than offence. ## quantitative analysis we first analyse the frequency distribution of each toxic/harmful label and its correlation with the epitome component levels. in addition, we inspect the difference between the empathetic toxic text and the non-empathetic toxic text. hereafter, for convenience, we will group the labels into three label groups based on their sources, which are e (epitome), p (perspective api), and o (openai moderation api). ## qualitative analysis to better understand the interplay between empathy and toxic/harmful characteristics, we selected a subset of the top 50 samples that exhibited high levels of empathy while also being associated with either of the toxic/harmful group labels (e∩(p ∪o)), and performed another manual annotation. here we collapsed the fine-grained labels of epitome and both apis categories into two binary labels "empathetic" and "toxic", and included a third class, "helpful" (also binary), to evaluate whether the responses has pragmatic benefit to the seekers. we define "helpful" as comments or content that have the intention or potential to help/improve the future situation or lessen the negativity of the seeker physically, mentally or emotionally. full definitions of all three classes are given in appendix d. the motivation for introducing the "helpful" class is to fill the gap in the current epinome annotations, which lack a metric for measuring the desired outcome or utility. in the context of mental health support, we propose the perception of "helpfulness" serves as a proxy for the desired outcome. for this exercise, the first author of this paper annotated all 50 samples. ## conclusion and limitation we examined the interplay of empathy and toxicity in responses to support seekers in mental health online discussions. our results found a mixed pattern of insignificant or weak (positive/negative) correlations between the epitome empathy labels and the toxic/harmful labels obtained from two widely used apis. we also revealed a significant presence of toxic/harmful content within empathetic instances in the epitome dataset, dominated by "profanity" and "self-harm" labels. these outcomes challenge the standard assumption that there is a negative correlation between empathetic and toxic/harmful language. interestingly, we found that the majority of empathetic toxic/harmful responses are not helpful for the individuals who are seeking help. we also noticed some well-intent responses being labelled as toxic due to use of profanity. these mislabels could stem from the issues of oversimplification and ambiguity in toxicity definitions, as previously highlighted in relevant studies @xcite . as argued by some communication studies @xcite (and also seen in our analyses), the utilization of toxic language does not invariably signify malicious intent. instead, it may function as a tool for emphasis, conveying closeness, or aligning with the conventions of a particular sociolect, or online context. this observation raises further questions about the role of domainand community-specific conceptualizations of toxicity in the realm of online content moderation. furthermore several empathetic instances are identified as containing suicidal ideation. this discovery raises concerns about the potential use of this dataset for empathetic fine-tuning purposes @xcite . to address these concerns, we recommend employing fine-grained toxicity detection models or apis for data filtering along with human manual validation to ensure alignment between the filtered data and the objectives of the fine-tuning task. we acknowledge a few limitations of this study: first, it only examines a single dataset within the mental health domain, and the predictions do not consider the context of the seeker's post due to api constraints. second, as demonstrated by both quantitative and qualitative validation of the apis' performance, the correspondence between the predicted toxic/harmful labels and human judgments is not perfect (though usable given the accuracy). third, the introduction of the "helpful" label in our analysis is a preliminary endeavor aimed at addressing the absence of a desired outcome metric in epitome, and as such is a (gross) simplification of the problem of measuring response utility. more refined measures, like empathic concerns (zahn-waxler and radke-yarrow, 1990) or selfreport surveys, might be worth considering in fu-ture studies. and lastly, the final manual annotation (emphathetic, toxic, and helpful) of the responses was done with a single annotator, and more thorough investigation is required to further validate the robustness of our findings. for future studies, we recommend a reevaluation and clarification of the role of empathy in text generation and understanding tasks. given that certain social science studies have indicated potential harm from empathetic behavior @xcite , further nlp research is needed to identify subcategories of empathy based on context that can either be beneficial or detrimental. finally, we suggest incorporating a measure of desired or undesired outcomes in future nlp studies, particularly when dealing with complex and sensitive concepts. this approach will facilitate the analysis and validation of the interplay between outcomes and mediating factors, such as empathy.