Measuring Geographic Performance Disparities of Offensive Language Classifiers (2022)

Abstract: Text classifiers are applied at scale in the form of one-size-fits-all solutions. Nevertheless, many studies show that classifiers are biased regarding different languages and dialects. When measuring and discovering these biases, some gaps present themselves and should be addressed. First, “Does language, dialect, and topical content vary across geographical regions?” and secondly “If there are differences across the regions, do they impact model performance?”. We introduce a novel dataset called GeoOLID with more than 14 thousand examples across 15 geographically and demographically diverse cities to address these questions. We perform a comprehensive analysis of geographical-related content and their impact on performance disparities of offensive language detection models. Overall, we find that current models do not generalize across locations. Likewise, we show that while offensive language models produce false positives on African American English, model performance is not correlated with each city’s minority population proportions. Warning: This paper contains offensive language.

## introduction many tasks revolving around text classification of social network data have been introduced including, but not limited to tracking viruses @xcite @xcite @xcite , providing help for (natural) disasters @xcite @xcite , detecting misinformation @xcite , and identifying cyberbullying @xcite . overall, text classifiers have been shown to be "accurate" across a wide range of applications. as deep learning models and packages have made substantial progress for the field of natural language processing (nlp), nlp models have become more accessible to the general public. hence, models are being deployed in a production environment and run at scale at a growing pace. however, recent work has shown that these models are biased and unfair, especially towards minority groups @xcite . in this paper, we expand on prior work by analyzing how model performance can fluctuate due do geographically-caused differences in language and topical content that exists in the context of offensive language detection. researchers have shown that topical and stylistic attributes of text are used by speakers on social media to implicitly mark their region-of-origin @xcite @xcite . for instance, @xcite show that doc2vec embedding frameworks can be leveraged to detect geolocation-related language differences. @xcite then introduces visualization techniques for measuring regional language change. @xcite shows that differences exist at the city level as well. hence, prior work has generally focused on incorporating or identifying regional aspects of language data to improve performance in machine translation @xcite or geolocation prediction and clustering @xcite . for particular downstream tasks, recent work in understanding performance disparities has found differences across various languages @xcite (e.g., finish vs. korean) and dialects @xcite -such as african american english (aae). likewise, @xcite and @xcite show that abusive and hate speech-related language classifiers are biased against aae-like text and machine learning models can learn these biases when certain populations are not being represented, making the data unbalanced. these results have been shown to extend into other text classifications tasks, for example, @xcite show that influenza detection models are also biased against aae-like text. similarly, @xcite find that part-of-speech tagging performance correlates with age. while there has been a substantial amount of research understanding, identifying, and measuring performance disparities across languages and dialects, to the best of our knowledge, there has been no prior work on measuring the performance of nlp classifiers across different geographic regions. specifically, prior work has not measured how geographical variations in language and topical content-or stance towards certain topics-impacts the performance of offensive language classifiers. complex interactions between topical content and style can impact model performance. even in the context of aae-related studies @xcite , aae is not spoken the same across different regions of the united states. there have been multiple studies in diversity, equity, and inclusion arguing against treating african americans as a monolithic group of people @xcite . moreover, certain features of aae only appear within specific regions of the us @xcite . likewise, geographic factors have been known to impact social behaviors such as voting turnout (zingher and moore, 2019) and general health disparities @xcite . hence, these geographic factors can impact both how people write and what people write about on social media. hence, to start addressing these issues, this paper proposes an initial study looking at how individual offensive language model's performance can vary geographically for the task of detecting offensive language due to the stylistic and topical differences in language. overall, to better understand the implications of geographical performance disparities offensive language models, we make three contributions: (1.) to the best of our knowledge, we perform the first analysis of geographical performance variation of offensive language classification models, producing novel insights and a discussion of important avenues of future research. ## language variation to the best of our knowledge, the impact geographical variation in language style and topical content has not yet been studied in the context of offensive language detection to the best of our knowledge. language variation is an important area of research for the nlp community. while there has been disagreement about whether morphology matters, @xcite has shown that incorporating information that can model morphological differences is important in improving model performance. prior work has generally focused on either developing methods to identify language features within text or use various language features to improve model performance. early work by @xcite showed that embeddings can capture geographically situated language, while doyle (2014) explored ways to quantify regional differences against a background distribution. recently, vardial has hosted an annual competition to identify various dialects of different languages (e.g., german and romanian) as well as geolocations @xcite . @xcite use topic distributions to show that different topics can provide signal to determine where the text originated from. for the same shared task, @xcite show that combining modern nlp architectures like bert with a double regression model can also provide success in determining the latitude and longitude points of the location for the given text. the results of this shared task highlights the fact that topical and lexical differences exist based on the location a tweet was written. other work around regional variation of language @xcite @xcite further prove that these differences in dialect and lexical patterns are significant across geographies. ## performance disparities performance disparities across languages and dialects recently have received attention in nlp. for example, recent research shows that performance drops in text classification models across different sub-populations such as gender, race, and minority dialects @xcite @xcite @xcite . @xcite al. (2019) and evaluate various methods such as word embedding debiasing and data augmentation to improve biased methods. @xcite shows that there is racial and ethnic bias when identifying hate speech online and show that tweets in the blackaligned corpus are more likely to get assigned as hate speech. overall, performance disparities have been observed across a wide array of nlp tasks such as detecting virus-related text @xcite , coreference resolution @xcite , named entity recognition @xcite , and machine translation (escudé font and costa-jussà, 2019). overall, the major gap in prior work investigating language variation is that there has not been any studies evaluating the impact regional language has on the performance of downstream tasks, particularly offensive language detection. hence, we measure performance disparities across geographical regions for the task of detecting offensive language. furthermore, many groups that are studied are "monolithic", such as male vs. female (using an unrealistic assumption of binary gender @xcite ), or aae which is not universally spoken in the same way within different cities in the us. for example, @xcite show that many well-known aae patterns (e.g., sholl, an nonstandard spelling of "sure") do not appear uniformly across the us. likewise, the discussion topics can also change regionally. hence, if an offensive language detection model performs poorly on one set of aae patterns or topics that only appear in a particular region, it can impact that location much more than others. hence, we believe that fine-grain regional analysis is a better future avenue to understand the real-world impact of nlp models. ## data collection and annotation in this section, we describe the two major datasets used in our experiments: the offensive language identification dataset (olid) @xcite and our newly constructed geographical diverse offensive language identification dataset (geoolid). a complete summary of the datasets can be found in olid. the olid dataset introduced by @xcite contains 14,100 tweets labeled to identify different levels of offensiveness including, but not limited to, not offensive, offensive, targeted offense, and not targeted offense. furthermore, targeted offenses are sub-categorized as targeting an individual, group, or other. for this study, we use the first level: not offensive (9,460 total) and offensive (4,640 total). ## experiments in order to address and test whether performance disparities exist across geographic regions for offensive language classifiers, we ran multiple experiments. we analyzed performance across the 15 cities in the geoolid dataset. in the following subsections, we provide the details of our experiments and provide evidence supporting that our geoolid dataset is representative of the unfiltered geoolid dataset and that offensive language classifier performance can vary by geolocation. in the final subsection, we explore the performance and language similarities across different geolocations that have similar demographics. ## conclusion we provide a comprehensive analysis of performance disparities of offensive language detection models. furthermore, we introduce a novel dataset that provides more than 14 thousand examples for further analysis of geographical differences in model performance. the study points to the importance of geographically sensitive nlp, where the impact and performance of nlp models are analyzed for specific geographical regions, or even micro communities within a city. moreover, finding regions where models perform poorly on can also provide unique testbeds as "hard test cases" similar to recent work on adversarial examples @xcite .