Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection (2023)

Abstract: The impact of AI models on marginalized communities has traditionally been measured by identifying performance differences between specified demographic subgroups. Though this approach aims to center vulnerable groups, it risks obscuring patterns of harm faced by intersectional subgroups or shared across multiple groups. To address this, we draw on theories of marginalization from disability studies and related disciplines, which state that people farther from the norm face greater adversity, to consider the “margins” in the domain of toxicity detection. We operationalize the “margins” of a dataset by employing outlier detection to identify text about people with demographic attributes distant from the “norm”. We find that model performance is consistently worse for demographic outliers, with mean squared error (MSE) between outliers and non-outliers up to 70.4% worse across toxicity types. It is also worse for text outliers, with a MSE up to 68.4% higher for outliers than non-outliers. We also find text and demographic outliers to be particularly susceptible to errors in the classification of severe toxicity and identity attacks. Compared to analysis of disparities using traditional demographic breakdowns, we find that our outlier analysis frequently surfaces greater harms faced by a larger, more intersectional group, which suggests that outlier analysis is particularly beneficial for identifying harms against those groups.

## introduction society often erects barriers that hinder marginalized individuals from receiving essential social and infrastructural access. disability studies offers valuable insights into their experiences, illuminating the oppressive and restrictive nature of the construction of "normalcy" within society. this rich body of literature highlights how prevailing social structures marginalize certain groups and further perpetuate their exclusion from mainstream societal participation @xcite . such arguments are also made in gender studies @xcite and related disciplines @xcite . extending the scope of these critiques to the realm of artificial intelligence, we recognize that ai models also encode prevailing notions of normalcy. when models optimize for an aggregate metric, they prioritize patterns and distributions for data points with more common characteristics; that is, the "norm" of the dataset. as such, individuals who fall outside the normative boundaries of the training data are more likely to be subject to model error and consequent harm. the lack of representation and tailored accommodation for these marginalized groups within the training data contributes to biased and exclusionary ai models. in the evolving landscape of powerful tools that use machine learning, it is crucial to critically evaluate their application to avoid reinforcing systemic biases and instead promote equitable outcomes. algorithmic auditing plays a vital role in assessing the real-world impact of ai, especially in identifying and scrutinizing potential harms to specific demographic subgroups and their intersections @xcite . however, the current subgroup-based analysis used in algorithmic fairness evaluations is fraught with challenges. two notable concerns emerge: first, identifying the relevant subgroups of concern is not always straightforward @xcite , as there may be hidden or overlooked patterns in how populations are affected by algorithmic harms. for example, dividing into individual racial subgroups may conceal shared harms experienced by multiracial individuals or people with different races. second, when considering intersectional subgroups across multiple demographic categories, including race, gender, sexual orientation, disability, religion, and age, the sheer number of potential subgroups becomes overwhelming while the size of each subgroup decreases; thus, particularly severe or unique harms faced by many smaller, intersectional subgroups may be overlooked @xcite . these limitations make it challenging to conduct thorough and accurate audits. inspired by the argument in disability studies that those who fall outside the norm experience greater adversity, we propose using outlier detection to statistically quantify who is assumed to be "normal" and analyze algorithmic harms with respect to this boundary in the domain of algorithmic content moderation. we find that disaggregating harm by outlier group membership reveals high model error disparities compared to other schemes for breaking down the data. additionally, we identify discrepancies in toxicity detection model performance between outliers and non-outliers and analyze these groups to determine the demographic makeup of those experiencing the most harm. this approach allows for the identification of subgroups and intersections thereof that are particularly vulnerable to harm in the model's behavior. this paper presents three primary contributions to advance the understanding of algorithmic harm towards marginalized groups in toxicity detection: our work underscores the importance of incorporating critical theory into auditing practices in machine learning models to ensure they do not exacerbate societal biases or inadvertently harm marginalized communities. the methodologies and tools we present serve as resources for those seeking to create more equitable and inclusive ai systems. ## prior work we ground our work on prior research on algorithmic content moderation and its potential harms, the measurement and evaluation of these harms, and the relationship between outlier detection and the social construction of the "other." ## methods current quantitative methods for determining the impact of content moderation systems on vulnerable populations rely on identifying demographic subgroups represented in the dataset and examining the model behavior toward these groups. however, this approach faces two primary barriers. first, dividing data by demographic group membership allows focus on model harm toward that group, but it can obscure insight into cross-group harm patterns. second, intersectional harms can be particularly acute and are crucial to measure. this has typically involved a disaggregated analysis of individuals along different demographic categories like race and gender @xcite . however, as more and more demographic categories are considered, the number of subgroups increases exponentially and their size decreases exponentially, making it challenging to identify and address specific problem areas. to mitigate the risk of missing significant harms, we propose a model harm identification framework to address these limitations and determine which groups and subgroups experience poorer model performance. ## results we used outlier detection to identify outliers on the basis of text, demographics, and annotator disagreement. approximately 1,000 samples were labeled as outliers for each outlier type since we set the contamination parameter in the local outlier factor (lof) algorithm to 0.05. we set the n_neighbors parameter to 4,000 based on the size of the dataset. we compared outliers and non-outliers by employing statistical testing to determine the significance of metric differences. we used the χ 2 test of homogeneity to compare group average scores, complemented with a bonferroni correction. ## conclusion our research leverages the insights of disability studies to illuminate unintended harms inflicted by ai systems, particularly those operating in toxicity detection. societal constructions of "normalcy"-often shaped and reinforced by statistical methodologies-can lead to biases and exclusions in ai models, which in turn amplify societal barriers and inequities. to address this challenge, we have made three contributions. first, we proposed and implemented a method for identifying marginalized groups at risk of ai harm by using outlier detection techniques to identify and examine three different types of outliers. second, we found that dividing the dataset by outlier status results in consistently high weighted differences in mse, indicating that our technique successfully exposes serious disparities in model error. finally, we critically examined model performance disparities across six types of toxicity, finding identity attacks and severe toxicity to be particularly acute and pervasive for demographic and text outliers. we also identified demographic groups as disproportionately represented among outliers, making them particularly vulnerable to harm from such disparities. there are two interesting side effects of these results. first, by determining which type of outlier causes the most difference, we can understand how the model responds to differences in syntactic (text features) and semantic (demographic mentions) information. second, if text-based outliers are insightful for uncovering model harm, model developers can allocate fewer resources to collecting high-quality demographic labels for their dataset, instead focusing on mitigating harms for groups overrepresented in text outliers. our research findings and methodologies pave the way for immediate application and future exploration in algorithmic auditing and harm mitigation. a promising direction for future research lies in assessing the use of outlier detection to identify algorithmically harmed groups in scenarios lacking explicit demographic data. this approach could broaden fairness auditing's scope and deepen our understanding of statistical normalcy, social marginalization, and their manifestations in ai systems.