A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by E nglish Marginal Abuse Models on T witter (2023)

Abstract: Harmful content detection models tend to have higher false positive rates for content from marginalized groups. In the context of marginal abuse modeling on Twitter, such disproportionate penalization poses the risk of reduced visibility, where marginalized communities lose the opportunity to voice their opinion on the platform. Current approaches to algorithmic harm mitigation, and bias detection for NLP models are often very ad hoc and subject to human bias. We make two main contributions in this paper. First, we design a novel methodology, which provides a principled approach to detecting and measuring the severity of potential harms associated with a text-based model. Second, we apply our methodology to audit Twitter’s English marginal abuse model, which is used for removing amplification eligibility of marginally abusive content. Without utilizing demographic labels or dialect classifiers, we are still able to detect and measure the severity of issues related to the over-penalization of the speech of marginalized communities, such as the use of reclaimed speech, counterspeech, and identity related terms. In order to mitigate the associated harms, we experiment with adding additional true negative examples and find that doing so provides improvements to our fairness metrics without large degradations in model performance.

## introduction because of the sheer volume of content, automatic content governance has been a crucial tool to avoid amplifying abusive content on twitter. harmful content detection models are used to reduce the amplification of harmful content online. these models are especially important to historically marginalized groups, who are more frequently the target of online harassment and hate speech @xcite . however, previous research indicates that these models often have higher false positive rates for marginalized communities, such as the black community, women, and the lgbtq community @xcite @xcite . within the context of social media, higher false positive rates for a specific subgroup pose the risk of reduced visibility, where the community loses the opportunity to voice their opinion on the platform. unfortunately, there are many contributing factors to over-penalization, including linguistic variation, sampling bias, annotator bias, label subjectivity, and modeling decisions @xcite @xcite . this type of over-penalization risks hurting the very communities content governance is meant to protect. algorithmic audits have become an important tool to surface these types of problems. however, determining the proper subgroups for analysis in global settings, and collecting high quality demographic information can be extremely challenging and pose the risk of misuse @xcite . current approaches to harm mitigation are often reactive and subject to human bias @xcite . in this work, we present a more principled and proactive approach to detecting and measuring the severity of potential harms associated with a text-based model, and conduct an audit of one of the english marginal abuse models used by twitter for preventing potentially harmful out-of-network recommendations. we develop a list of keywords for evaluation by analyzing the text of previous false positives to understand trends in the model's errors. this allows us to alleviate concerns of false positive bias in content concerning or created by marginalized groups without using demographic data. ## conclusion current approaches to harm mitigation and bias detection are frequently reactive and subject to human bias. additionally, demographic labels and dialect classifier are difficult to acquire and pose