Unveiling Identity Biases in Toxicity Detection : A Game-Focused Dataset and Reactivity Analysis Approach (2023)

Abstract: Identity biases arise commonly from annotated datasets, can be propagated in language models and can cause further harm to marginal groups. Existing bias benchmarking datasets are mainly focused on gender or racial biases and are made to pinpoint which class the model is biased towards. They also are not designed for the gaming industry, a concern for models built for toxicity detection in videogames’ chat. We propose a dataset and a method to highlight oversensitive terms using reactivity analysis and the model’s performance. We test our dataset against ToxBuster, a language model developed by Ubisoft fine-tuned for toxicity detection on multiplayer videogame’s written chat, and Perspective API. We find that these toxicity models often automatically tag terms related to a community’s identity as toxic, which prevents members of already marginalized groups to make their presence known or have a mature / normal conversation. Through this process, we have generated an interesting list of terms that trigger the models to varying degrees, along with insights on establishing a baseline through human annotations.

## introduction online spaces are valuable for exchanging ideas and discussing common interests globally. however, these interactions are often marred by toxic comments and content, evident on platforms like facebook @xcite , twitter @xcite , and reddit @xcite . the videogame industry is also not immune to harm and harassment, as evidenced by the rising toxicity in written communications among players @xcite . this high level of toxicity not only affects gaming choices but also the personal lives of players involved. consequently, platforms (hanu and @xcite and the videogame industry @xcite @xcite have turned towards language models for toxicity detection and content moderation due to their excellent performance and contextual understanding. although these models can effectively capture toxic content, they can perpetuate and amplify social biases present in their training datasets @xcite @xcite . biases can emerge during dataset creation when practitioners sample data, annotators label data based on personal understanding, culture, and experiences, and practitioners aggregate labels. in this study, we specifically focus on the issue of models overestimating the toxicity of terms associated with certain concepts, leading to problematic false positives and even false negatives @xcite @xcite @xcite . existing research has primarily focused on biases in toxicity detection without considering the specific use-case of in-game chat, despite the widespread presence of toxicity in that particular context. to fill this research gap, we begin by meticulously constructing a dataset that aims to uncover identity biases present in language models. this dataset encompasses biases commonly observed within the english-speaking gaming community in north america. next, we introduce a novel approach that combines reactivity analysis and the model's performance to identify highly sensitive terms. we apply this method to assess the effectiveness of toxbuster @xcite , a model specifically trained for in-game chat, as well as perspective api. through our evaluation, we demonstrate the efficacy of our approach and its potential for evaluating various forms of biases in toxicity detection models. both the dataset and the method are described in this paper as a proof of concept, as they do not cover every possible bias. additional iterations are possible and strongly encouraged. in summary, our contributions are two-fold, presenting a prototype of the following: ## related work toxicity detection is inherently complex and subjective, with different definitions and interpretations among researchers @xcite . biases also vary across communities, influenced by culture, origin and socio-political context. in this study, we define biases as "prejudice in favour of or against one thing, person, or group compared with another usually in a way that's considered to be unfair" (university of california). natural language processing encompasses a wide range of types of biases, categorized by their sources or the type of harm they cause @xcite . our focus lies specifically on lexical identity biases, which refer to biases conveyed by terms related to one's identity or characteristic @xcite . initially, we address the questions posed by blodgett et al. (2020), more precisely "what kinds of system behaviours are harmful, in what ways, to whom, and why". the harmful behaviour we examine are false positives and false negatives concerning identity biases. in other words, we aim to identify terms that the model consistently tags as toxic or non-toxic, even when they are used in the opposite manner. a false positive resulting from an identity bias prevents marginalized and possibly minority communities from discussing and engaging with members of their own group @xcite thereby erasing proper representation of that social group @xcite . conversely, a false negative neglects to flag a sentence containing a term that should be identified as toxic, usually associated with an oppressing or majority community. previous work on bias detection has focused on creating evaluation metrics @xcite or corpora @xcite that consider binary dimensions, such as gender (male, female) or race (black, white), to determine the model's leanings. however, online spaces such as videogames encompass diverse communities that cannot be easily classified into such binary labels, rendering this analysis insufficient. @xcite developed a dataset of sentence templates and human names to identify biases in a model, however these longer sentences and names are not typically found in videogame chat. ## methodology here, we present our method for identifying biases and oversensitive terms toxicity models flag in four sections. section 3.1 discusses the generation of a synthetic dataset using sentence templates and a set of manually identified terms associated with potential biases. section 3.2 outlines the annotation process for this dataset. section 3.3 focuses on using reactivity analysis to highlight terms and biases. lastly, in section 3.4, we describe the two toxicity detection models that we measure. ## results and discussion after obtaining all the annotations and predictions, we calculate the proportion of toxic labels for each source, as depicted in right away, we notice a disparity in toxic label proportions across annotations and the two toxicity detection models, suggesting the presence of biases and a varying effectiveness. it prompts a closer examination of the model's performance, which will be detailed in the next sections. we first performed the reactivity analysis on the models and the human annotations propagation. the precision, recall and f1 scores are used to evaluate the models' predictions on the dataset, using the propagation as a gold label. ## conclusion we have developed an evaluation dataset that includes examples of in-game chat lines with a range of terms related to identity biases. our approach proposes a novel method that utilizes reactivity analysis and model performance to identify sensitive terms with biases in toxicity detection models and would need further human interventions. we have applied this analysis to toxbuster and perspective api, demonstrating the potential application of our method and dataset to models beyond the gaming domain. through this process, we have generated an interesting list of terms that trigger the models to varying degrees, along with insights on establishing a baseline through human annotations. these findings can contribute to providing explanations for the models' predictions, which is crucial in bias and fairness research. we now have a clearer roadmap for future steps, including obtaining a reliable ground truth through diverse human annotators, evaluating the models with different settings and parameters, and incorporating linguistic and sociolinguistic considerations to enhance our understanding of how these terms operate in both gaming and non-gaming contexts.