Enriching Abusive Language Detection with Community Context (2022)

Abstract: Uses of pejorative expressions can be benign or actively empowering. When models for abuse detection misclassify these expressions as derogatory, they inadvertently censor productive conversations held by marginalized groups. One way to engage with non-dominant perspectives is to add context around conversations. Previous research has leveraged user- and thread-level features, but it often neglects the spaces within which productive conversations take place. Our paper highlights how community context can improve classification outcomes in abusive language detection. We make two main contributions to this end. First, we demonstrate that online communities cluster by the nature of their support towards victims of abuse. Second, we establish how community context improves accuracy and reduces the false positive rates of state-of-the-art abusive language classifiers. These findings suggest a promising direction for context-aware models in abusive language research.

## introduction existing models for abuse detection struggle to grasp subtle knowledge about the social environments that they operate within. they do not perform natural language understanding and consequently cannot generalize when tested out-ofdistribution @xcite . this problem is often the result of training data imbalance, which encourages language models to overestimate the significance of certain lexical cues. for instance, @xcite observe that "commentator", "football", and "announcer" end up strongly correlated with hateful tweets in the @xcite corpus. this trend is caused by focused sampling, and it does not reflect an underlying property of abusive expressions. when models rely on pejorative or demographic words, they can encode systemic bias through false † these authors made equal contributions. productive conversations containing slurs are common, and they take many forms @xcite . research inspired by the #metoo movement has focused on the detection of sexual harassment disclosures by victims @xcite , but this research has not been sufficiently integrated into the literature on abusive language detection. the distinction between actual sexist messages and messages calling out sexism is rarely addressed in the field @xcite . a similar trend is seen with sarcasm. humor and selfirony can be employed as coping mechanisms by victims of abuse @xcite ), yet they constitute frequent sources of error for state-of-theart classifiers @xcite . for example, the median toxicity score for language on transgendercirclejerk, a "parody [subreddit] for trans people", is as high as 90% @xcite . more broadly, transgender users are "excluded, harmed, and misrepresented in existing platforms, algorithms, and research methods" related to network analysis @xcite . meaningful improvements in abusive language detection require a thoughtful engagement with the perspectives of marginalized communities and their allies. one way to ensure that machine learning frameworks are socially conscientious is to add context around conversations. past research has explored the contextual information within conversation threads @xcite , user demographics (unsvåg and @xcite , user history @xcite @xcite , user profiles (unsvåg and @xcite , and user networks @xcite with varying degrees of success in improving performance. however, most modelling efforts for abusive language detection neglect one major aspect of online conversations: the community environment they take place within. online communities adhere to a variety of sociologistic norms that reinforce their identities. this phenomenon is easily observed on reddit, where community structure is an explicit component of platform design. for example, the majority of comments on the pro-trump subreddit the_donald delegitimize liberal ideas (mclamore and ulug, 2020; @xcite . similarly, a collection of "manosphere" subreddits espouse misogynistic ideologies @xcite . more broadly, communities can reinforce "toxic technocultures" @xcite , and those technocultures are not limited to reddit. community structure is present across 4chan, facebook, voat, etc., and it exists in a less explicit manner on platforms like twitter @xcite . in this paper, we study community context on reddit, and we focus specifically on language that is collected using slurs. we demonstrate that subreddits cluster by the nature of their support towards marginalized groups, and we use subreddit embeddings to improve the accuracy and false positive rates of state-of-the-art abusive language classifiers. while our analysis is platform-specific, it suggests a promising new direction for contextaware models. ## discussion our analysis points to two key resources that would benefit future abusive language research. ## conclusion and future work the subjectivity of abuse makes it challenging to annotate and detect reliably. one method for making the problem tractable is to position online conversations within a larger context. this paper was an exploration of one type of contextual in-formation: community identity. we found that the context derived from community identity can help in the collection and classification of abusive language. we therefore believe that community context is integral to all stages of abusive language research. we leave as future work the inclusion of community information in existing, platformagnostic, ensemble detection frameworks.