Modeling Users and Online Communities for Abuse Detection: A Position on Ethics and Explainability (2021)

Abstract: Abuse on the Internet is an important societal problem of our time. Millions of Internet users face harassment, racism, personal attacks, and other types of abuse across various platforms. The psychological effects of abuse on individuals can be profound and lasting. Consequently, over the past few years, there has been a substantial research effort towards automated abusive language detection in the field of NLP. In this position paper, we discuss the role that modeling of users and online communities plays in abuse detection. Specifically, we review and analyze the state of the art methods that leverage user or community information to enhance the understanding and detection of abusive language. We then explore the ethical challenges of incorporating user and community information, laying out considerations to guide future research. Finally, we address the topic of explainability in abusive language detection, proposing properties that an explainable method should aim to exhibit. We describe how user and community information can facilitate the realization of these properties and discuss the effective operationalization of explainability in view of the properties.

## introduction with the advent of social media, anti-social and abusive behavior has become a prominent occurrence online. undesirable psychological effects of abuse on individuals make it an important societal problem of our time. @xcite studied the ill-effects of online abuse on children, concluding that children may develop depression, anxiety, and other mental health problems as a result of their encounters online. pew research center, in its latest report on online harassment @xcite , revealed that 40% of adults in the united states have experienced abusive behavior online, of which 18% have faced severe forms of harassment, e.g., that of sexual nature. these statistics stress the need for automated detection and moderation systems. hence, in recent years, a new research effort on abusive language detection has sprung up in nlp. that said, the notion of abuse has proven elusive and difficult to formalize. different norms across different (online) platforms can affect what is considered abusive @xcite . in the context of natural language, abuse is a term that encompasses many different fine-grained types of negative expressions. for example, @xcite use it to collectively refer to hate speech, derogatory language and profanity, while @xcite use it to discuss racism and sexism. the definitions for different types of abuse tend to be overlapping and ambiguous. however, regardless of the specific type, we define abuse as any expression that is meant to denigrate or offend a particular person or group. taking a coursegrained view, @xcite classify abuse into broad categories based on explicitness and directness. explicit abuse comes in the form of expletives, derogatory words or threats, while implicit abuse has a more subtle appearance characterized by the presence of ambiguous terms and figures of speech such as metaphor or sarcasm. directed abuse targets a particular individual as opposed to generalized abuse which is aimed at a larger group such as a particular gender or ethnicity. to date, several approaches to automated detection of abusive language have been proposed, including rule-based @xcite @xcite , linguistic and social feature engineering @xcite @xcite , utilizing distributed representations from neural networks @xcite @xcite or applying deep neural networks directly @xcite @xcite . researchers have also explored multi-task learning settings with objectives such as emotion detection @xcite . we refer the reader to recent surveys of the field @xcite for a detailed literature review. more recently, researchers have noted that the linguistic features of a comment alone may not be sufficient to classify it as abusive or not. information of the user who posted the comment, and of the surrounding social community of that user, further provides valuable insights into the abusiveness of the comment. an example of this is the study by @xcite , which mapped the locations of racist tweets in response to president obama's re-election to show that such tweets were not uniformly distributed across the united states but instead came from specific geographical communities of users. other works have also shown how users on online platforms organize into communities based on factors such as shared beliefs, stereotypes, linguistic norms, or geographical propinquity @xcite . in this paper, we focus on the role that modeling of users and communities plays in the automated detection of abusive language on online platforms. specifically, we investigate the different state of the art methods that leverage user or community information to enhance the understanding and detection of abusive language. while these methods have yielded high performance gains, there has been little discussion of the kinds of information they capture. we provide a comprehensive review of these methods, analyzing the information they encode about users or communities and the relevance of that for detection of abusive language. we then explore the ethical considerations of incorporating user and community information in such methods, providing guidance for future research. finally, we address the topic of explainability in abusive language detection, proposing properties that an explainable detection method should aim to exhibit. we describe how user and community information can facilitate the realization of these properties and discuss the effective operationalization of explainability in view of the properties. ## why the user and community matter throughout the paper, user refers to the user of an online platform who may have posted a comment that is to be classified as abusive or not. the community of this user comprises other users and contents that they interact with on the online plat-form. in other words, community refers to the neighborhood of the user in the social graph of the platform. conversations online are inherently contextual. consequently, abuse on online platforms can only be effectively interpreted within a larger context @xcite rather than in isolation. this is especially true for implicit or generalized abuse, which are harder to interpret than explicit abuse for humans and machines alike. information of the user who posted the comment, or of the surrounding community including the targets of the comment, offers insights into several aspects of the context that are otherwise not accessible through the linguistic content of the comment alone. here, information may refer to demographic traits like age or gender, knowledge about linguistic behavior, location details, etc. below we categorize and discuss the aspects of the context relevant to abusive language detection. ## modeling the user and community in this section, we first recap the datasets in the domain of abusive language detection that contain user or community information alongside comments. we then go on to discuss the methods that have been applied to them. ## analysis of the methods we now analyze the methods described above to understand the gains that user or community information provides. based on this analysis, in the next sections, we explore the ethical considerations of incorporating user and community information and how it can support explainability. across the three categories of methods, we note that the general setup is to create representations, called profiles, for users or communities and utilize them alongside linguistic features. in social feature engineering based methods, these profiles are manually constructed vectors of features that capture the relevant traits, such as age in the case of cyber-bullying and gender in the case of sexism. in user embeddings and social graph based methods, the profiles are instead generated by neural network architectures to capture the linguistic behavior or community traits of users. that said, across all three categories, the profiles essentially provide a wider context to the comment being classified for abuse. for example, having the gender of the user who produces a comment such as "had an accident, women can't drive it seems!" can help to classify the comment as sexist or not by differentiating benign self-deprecating humor from intent to degrade. the context that the profiles encode increases as we go from social feature engineering based methods to user embeddings based methods and further to social graph based methods. this is also evident from the magnitude of gains that the profiles provide on top of linguistic features. for example, the gender feature only increases the f 1 from 73.89% to 73.93% over character n-gram counts on the dataset by @xcite , while the social graph based method of @xcite increases the f 1 to above 80%. the example aside, it makes intuitive sense that profiles from social graph based methods encode the most amount of context, since these profiles are able to capture the various phenomena that occur in social networks, the most prominent ones of which are: in fact, homophily is so prominent, @xcite noted in their work that the profiles they generated from the social graph of users and tweets could encode patters of similar linguistic practices amongst connected users in the waseem and hovy (2016) dataset, hence allowing for comments with implicit and generalized sexism or racism to be better detected. moreover, homophily has direct associations with all the four aspects of context that we described in section 2, i.e., similar sociolinguistic norms and shared language markers facilitate homophilic ties in social networks @xcite , as do shared beliefs, stereotypes, and demographic traits @xcite . therefore, capturing homophily allows for all the four aspects to be directly captured together. we note that just exploiting simplistic and limited inductive biases that are easy to extract, like gender of the user, can render methods prone to making faulty generalizations because of overfitting to patterns in the training data. this is also evident from the observations that @xcite made in their work. they noted that the profiles they generated from the social graph consisting of user and tweet nodes improved f 1 scores over the profiles @xcite generated from the social graph just consisting of users, with the gains mainly coming from increase in precision. it is because solely relying on network homophily as the inductive bias for generating profiles caused the method of @xcite to make some faulty generalizations. such observations have also been made by other works, a prominent one of which is the work of @xcite who explored the relationships amongst gender, language, and social network connections. the researchers noted that even though there may exist many linguistic clusters that exhibit strong orientations to one gender, yet the characteristics of any particular cluster do not necessarily align with populationlevel statistics for that gender. furthermore, they observed that there are individuals whose linguistic practices differ from population-level trends for their gender and that gender homophily does not capture their linguistic practices. ## ethical considerations while researchers have started incorporating user and community information into detection of abusive language, there has been no discussion of the ethical guidelines for doing so. therefore, taking a stand on the issue, we lay out five ethical considerations in the design and implementation of methods that incorporate user or community information: personal vs. population-level trends. it is important to perform appropriate generalizations from personal traits to population-level behavioral trends. methods should avoid relying on simple inductive biases such as personal traits of users, e.g., gender, race, etc., as this can easily lead to scenarios of faulty generalizations where comments from a particular gender or race are always labeled abusive/benign. moreover, relying solely on personal traits of users also comes with the risk that such information may not always be present or may not be accurate even when present @xcite . on the other hand, more complex inductive biases learned from data, as in the case of social graph based methods, provide a safer and more reliable generalization from personal behaviors of users or communities to population level trends. ## explainable abusive language detection explainability is an important concept within abusive language detection. @xcite noted in their work that explainable ml techniques can promote restorative and procedural justice by surfacing the norms that have been violated and clarifying how they have been violated. that said, there has been limited discussion of the issue within the domain of abusive language detection. in this section, we first formalize the properties that an explainable detection method should aim to exhibit in order to thoroughly substantiate its decisions. we then describe how user and community information play an important role in the realization of each of the properties. finally, we discuss what it means to operationalize explainability within abusive language detection in an effective manner. ## conclusions abuse on the internet stands as a significant challenge before the society. its nature and characteristics constantly evolve, making it a complex phenomenon to study and model. in this paper, we explored the ways in which users and communities play a role in the detection of abusive language. we investigated the methods that leverage user or community information to uncover how they work and the knowledge they capture. we then explored the ethical challenges of incorporating user and community information, laying out considerations to guide future research. finally, we moved to the topic of explainability in abusive language detection, proposing properties that an explainable detection method should aim to exhibit. we describe how user and community information can facilitate the realization of these properties and discussed the effective operationalization of explainability in view of the properties.