Giving Control Back to Models: Enabling Offensive Language Detection Models to Autonomously Identify and Mitigate Biases (2024)

Abstract: The rapid development of social media has led to an increase in online harassment and offensive speech, posing significant challenges for effective content moderation. Existing automated detection models often exhibit a bias towards predicting offensive speech based on specific vocabulary, which not only compromises model fairness but also potentially exacerbates biases against vulnerable and minority groups. Addressing these issues, this paper proposes a bias self-awareness and data self-iteration framework for mitigating model biases. This framework aims to “giving control back to models: enabling offensive language detection models to autonomously identify and mitigate biases” through bias self-awareness algorithms and self-iterative data augmentation method. Experimental results demonstrate that the proposed framework effectively reduces the false positive rate of models in both in-distribution and out-of-distribution tests, enhances model accuracy and fairness, and shows promising performance improvements in detecting offensive speech on larger-scale datasets.

## introduction the rapid development of social media has significantly enhanced the ease with which people can connect, share, and obtain data online, as well as convey emotional messages. however, the convenience of internet technology has concurrently increased the risk of individuals encountering cyberbullying and online attacks. automatic detection of offensive language is an effective measure to maintain the safety, health, and friendliness of online social platforms @xcite . this technology has broad applications across various internet interaction environments, including social networks, online forums, instant messaging * equal contribution † corresponding author: fxc1982@xjnu.edu.cn tools, news media platforms, and gaming communities. by integrating multiple natural language processing (nlp) techniques, numerous models @xcite @xcite have been designed and applied to the task of detecting offensive language. however, even the most advanced models tend to overly rely on specific words to predict offensive content @xcite , often mistakenly classifying sentences containing these words as offensive @xcite . this phenomenon raises concerns about bias in offensive language detection systems, thereby limiting their fairness @xcite . additionally, it can lead to prejudiced treatment of vulnerable and minority groups, potentially exacerbating racism @xcite . in offensive language detection, not only identity-related vocabulary such as "gay" or "black" @xcite but also non-identityrelated vocabulary like "sport" and "football" are often inappropriately associated with offensive content. one of the root causes of this issue lies in the biases present in the data collection process @xcite . because the collected data frequently place these specific vocabulary in offensive contexts, it fosters erroneous statistical associations between these vocabulary and offensive labels, known as spurious statistical correlations. models learn and make predictions based on these spurious statistical correlations, leading to biases in the models themselves. these incorrectly associated vocabulary are commonly referred to as "spurious artifacts" and their associations with labels are termed "spurious correlations" @xcite . regarding the identification of spurious artifacts, @xcite approached this issue by examining datasets and employing statistical methods such as pointwise mutual information (pmi) to measure the potential association strength between a word and offensive labels. subsequently, they used manual annotation to identify spurious artifacts. however, this method has two significant drawbacks: 1) given the vastness of datasets, manual annotation is impractical. 2) the spurious artifacts identified from the dataset may not be universally applicable to all models; for instance, model a might be misled by a spurious artifact x, while model b remains unaffected. to mitigate model biases, @xcite proposed a data augmentation method that utilizes large language models(llm) like gpt-3 to generate sentences and expand negative sample instances, thereby balancing the dataset and reducing model bias. experimental results indicate that data augmentation is an effective approach for mitigating model bias. however, determining the amount of data augmentation often relies on the researchers' prior experience and lacks objective criteria, making the process largely subjective. to address the aforementioned issues, we propose a model bias correction framework based on bias self-awareness and data self-iteration (bsadsi), which is founded on the core principle of "giving control back to models". the bsadsi framework incorporates an innovative model bias self-awareness algorithm (mbsa), enabling the model to autonomously identify and acquire spurious artifacts. furthermore, bsadsi integrates reinforcement learning strategies, allowing the model to independently determine the content and extent of data augmentation. our main contributions are as follows: ## related work in this chapter, we systematically review research findings in two aspects: identifying spurious correlations in detecting offensive language and methods for mitigating model biases. ## methodology the model bias correction framework bsadsi we proposed is illustrated in figure 1 . this framework primarily consists of two processes: model bias self-awareness (mbsa) and self-iterative data augmentation method. in the mbsa process, we initially use an offensive speech detection model to classify the data from the validation set, identifying instances with high confidence but incorrect judgments to construct a bias dataset. subsequently, we extract vocabulary from this bias dataset, conduct filtering and validation to obtain a set of spurious artifacts. finally, we compute a bias coefficient for each spurious artifacts to determine the scale of generated data. during the self-iterative data augmentation process, we introduce reinforcement learning strategies where the offensive speech detection model acts as an agent. through interactions between mbsa and a reward function feedback loop, the large language model iteratively generates sample data containing spurious artifacts, thus expanding the training set contrapuntally. this iterative process dynamically adjusts the quantity of newly added data, optimizing the model's ability to dentify and correct biases. algorithm 1 outlines the iterative process of the bsabsi framework. initially, the model undergoes initial fine-tuning on the unaugmented base dataset. subsequently, the model's performance is evaluated using a reward function, recording this initial score. the mbsa module analyzes the spurious artifacts set generated by the model in this round and determines the demand for negative example samples. this information guides the large language model to generate negative example samples, which are then integrated into the training dataset, completing the initial augmentation.as the process proceeds to the n-th iteration, the model undergoes further fine-tuning on the dataset expanded from the previous n-1 rounds. after adjustments, the model is re-evaluated using the scorer, comparing its score with that of the n-1 rounds. if no score improvement is observed for t consec-algorithm 1: iterative model refinement with bsabsi result: model refined with iterative data augmentation until optimal. input :original dataset d, scoring function r, mbsa module, language generation module for negative samples. utive rounds, the model is deemed optimal, and the iteration process terminates. conversely, if performance continues to improve, mbsa intervenes again to analyze the spurious artifacts set identified by the model in this round and determine the scale of additional negative example samples to be added.it is noteworthy that if mbsa in a particular round fails to discover new spurious artifacts, the iteration will also terminate. if the termination condition is not met, the iterative process described above is repeated. ## experiment and analysis in this section, we first introduce the dataset, model and evaluation metrics. next, we compare the model after correction with the uncorrected model using bsadsi. finally, detailed analysis is provided. model spurious artifacts bert 日本(japan), 外地人(outsider),中国(china), 白人(whites), 四川人(sichuanese) roberta 暴 力(violence), 男 人(man), 素 质(moral quality), 反感(dislike) ## conclusion the bsadsi framework we proposed demonstrates significant effectiveness in mitigating biases in offensive speech detection models. at its core, this framework aims to give control back to the model itself to correct biases by employing bias self-awareness algorithms and self-iterative data augmentation method. the bias self-awareness algorithm automates bias data acquisition, identifies spurious artifacts, and calculates bias coefficients, thereby enhancing efficiency in recognizing spurious associations and ensuring that the model can identify and understand the sources of bias based on its own characteristics. the self-iterative data augmentation method introduces reinforcement learning strategies, allowing the model to autonomously determine the content and scale of data expansion based on feedback from mbsa, thereby achieving dynamic optimization of data augmentation. experimental results indicate that the bsadsi framework not only effectively reduces the false positive rate in both in-distribution and out-of-distribution tests but also enhances model accuracy and fairness. moreover, it shows promising potential to significantly improve the performance of offensive speech detection on largerscale datasets.