Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021)

Abstract: Online social media platforms increasingly rely on Natural Language Processing (NLP) techniques to detect abusive content at scale in order to mitigate the harms it causes to their users. However, these techniques suffer from various sampling and association biases present in training data, often resulting in sub-par performance on content relevant to marginalized groups, potentially furthering disproportionate harms towards them. Studies on such biases so far have focused on only a handful of axes of disparities and subgroups that have annotations/lexicons available. Consequently, biases concerning non-Western contexts are largely ignored in the literature. In this paper, we introduce a weakly supervised method to robustly detect lexical biases in broader geo-cultural contexts. Through a case study on a publicly available toxicity detection model, we demonstrate that our method identifies salient groups of cross-geographic errors, and, in a follow up, demonstrate that these groupings reflect human judgments of offensive and inoffensive language in those geographic contexts. We also conduct analysis of a model trained on a dataset with ground truth labels to better understand these biases, and present preliminary mitigation experiments.

## introduction online social media platforms have increasingly turned to nlp techniques to detect offensive and abusive content online at scale (e.g., the perspective api), as a way to mitigate the harms it causes to people. 1 however, recent research has shown that these models often encode various societal biases against marginalized groups @xcite @xcite @xcite @xcite @xcite and can be adversarially deceived in this paper, we propose a new weakly supervised method (outlined in figure 1 ) to address shortcomings of current bias detection methods and robustly detect lexical biases in models (i.e. biases that associate the presence or lack of toxicity with certain words) when applied to noisy text (in particular, text generated by social media users from non-predominantly anglophone nations). we draw upon the observation that social stereotypes and biases in data often reflect the public discourse at a specific point in time and place @xcite . we use a crosscorpora analysis across seven different countries to reveal terms overrepresented in each country. we then cluster the model behaviour against term perturbations to qualify these biases further. furthermore, we demonstrate the effectiveness of our method through human validation. our method is task and genre agnostic; we expect it to work in any culture-laden tasks on genres of noisy text from platforms with global presence. moreover, it can be applied in cases with and without ground truth labels. we conduct experiments on both cases and present a preliminary analysis on mitigation strategies to reduce harms created by these biases. ## background current research on detecting biases in nlp models primarily starts with a specific axis of injustice, and relies on annotated data @xcite or lexicons that are salient to different subgroups. for example, @xcite use a list of lgbtq identity terms to identify varied outcomes in text classification and @xcite use a list of terms that refer to persons with a disability. however, this reliance on lexicons and annotated data limit the focus to a handful of social injustices for which resources/lexicons are available, and largely ignores biases in non-western contexts @xcite and online environments containing creative or noisy language variation. moreover, prior work shows that interpretation of hate varies significantly across different cultural contexts @xcite and reveals that the performance state-of-theart models such as bert @xcite on various tasks degrades significantly when used on noisy text @xcite . consequently, current approaches to identifying model biases miss 1) undesirable biases around words that are not captured in existing lexicons/datasets, and 2) desirable biases around offensive terms that are salient to different geocultural contexts that the models fail to capture. as an example, such biases are present even in a commonly-used toxicity detector like the perspective api, as shown in ## methodology we find undesirable lexical biases in two phases. the first phase aims to find words that are statistically overrepresented in tweets from specific countries that were deemed toxic by the model. such overrepresentation could be due to different reasons: (1) the model has learned some of these words to be offensive and hence their presence causes it to assign a higher toxicity score, or (2) some of the words appear more often in offensive contexts (e.g., frequent victims of toxicity) but their presence is merely correlational (i.e., the model doesn't have any toxicity association for these words). another consideration is whether the association of a particular word towards toxicity is desirable or not. for some words, such as country-specific slur words, we want the model to be sensitive to their presence. on the other hand, we desire the models to not have biases towards culturally salient non-offensive words, such as religious concepts. the second phase attempts to separate the overrepresented words along these two dimensions: our method aims to balance (i) achieving reasonably high precision and (ii) narrowing down the search space of words enough to be feasible in phase 2 and any further human evaluation studies. we now discuss the methods we use for each phase. ## evaluation of off-the-shelf models our approach can notably be applied to cases where ground truth labels are not available. this is useful in deployment settings, where it may not be feasible and/or be expensive to annotate sufficient amounts of data regularly or where one does not have access to the training data but wants to understand biased outcomes. we provide an example of how our method could be used in this setting. ## towards mitigating undesirable biases our pipeline provides an unsupervised approach to identifying potential latent biases in toxicity models. ideally, practitioners aim not only to identify these biases but also to fix them before deploying their systems. however, our method identifies terms of different types: causal-desirable associations and correlational-undesirable associations, as well as causal-undesirable and correlationaldesirable associations both of which require different approaches to handle. the clusters we find in phase 2 approximate these different types of associations, and hence we expect them to behave differently for different mitigation approaches. here, we test the impact of baseline mitigation techniques on reducing bias in these clusters. ## discussion and conclusion toxicity detection models commonly rely on training data that reflects discourse that is temporally and culturally situated. as such, models may contain biases by learning undesirable features that are not toxic or be blind to toxic language underrepresented in their training data. here, we introduce a new weakly supervised approach for uncovering these biases. 7 our method provides a key diagnostic tool for model creators and deployers to test for biases. from a fairness perspective, our analysis reveals biases present in the model along different dimensions, without having any labeled resources for those dimensions. for instance, c0 revealed biases around religion (muslim, hindu, jews), country/ethnicity (indian, pakistani, arabs), and ideology (feminist, liberals, communist) without looking for these axes a priori. this could be a valuable first step before deeper analysis of these biases. at a practical level, our method reveals countryspecific offensive terms that the model has not seen before (e.g., maadarchod). such country-specific abusive language lexicons could aid nlp practitioners trying to make their models robust across geographies. such lexicons could also aid humanrights organizations working on building regionspecific hate-speech lexicons. 8 furthermore, our model also identifies portmanteau (blended) words that are used disproportionately in certain geocultural contexts (e.g., presstitute, and libtard in 7 all code and data to replicate the results will be released at the url: https://github.com/sghosh73/ cross-cultural-bias-detection 8 https://www.peacetechlab.org/ indian social media discourse). our method could help monitor emergent offensive language in different geographies and aid data augmentation efforts in keeping models up-to-date through surfacing emergent language in new geographies.