Are you sure? Measuring models bias in content moderation through uncertainty (2025)

Abstract: Automatic content moderation is crucial to ensuring safety in social media. Language Model-based classifiers are increasingly adopted for this task, but it has been shown that they perpetuate racial and social biases. Even if several resources and benchmark corpora have been developed to challenge this issue, measuring the fairness of models in content moderation remains an open issue. In this work, we present an unsupervised approach that benchmarks models on the basis of their uncertainty in classifying messages annotated by people belonging to vulnerable groups. We use uncertainty, computed by means of the conformal prediction technique, as a proxy to analyze the bias of 11 models (LMs and LLMs) against women and non-white annotators and observe to what extent it diverges from metrics based on performance, such as the F1 score. The results show that some pre-trained models predict with high accuracy the labels coming from minority groups, even if the confidence in their prediction is low. Therefore, by measuring the confidence of models, we are able to see which groups of annotators are better represented in pre-trained models and lead the debiasing process of these models before their effective use.

## introduction language models (lms), including large language models (llms), have been widely used in realworld applications since their introduction due to their ability to generate and understand human-like text and have been adopted as street-level algorithms @xcite : technologies that are implemented to enforce the rules of platforms based on user-generated content. in such a sense, these technologies play the role of bureaucrats, as they provide an interpretation of community guidelines and decide accordingly which users' behaviors must be removed and which not. streetlevel algorithms are widely present in social media @xcite , but their adoption is not limited to this context. toxicity classifiers such as the perspective api @xcite are used to filter out unwanted texts from pretraining data, playing a crucial role in the development of fair models. @xcite highlights the over-reliance on data of these technologies, leaving the power of what the models know and how to address the problems to those who produce and maintain the data. the societal cost of doing that is high, and it has been demonstrated that such technologies tend to perpetuate social biases against vulnerable minorities, whose opinion is less represented or excluded in data @xcite . in this paper, we investigate the presence of biases towards underrepresented groups in hate speech detection, using two datasets, social bias inference corpus @xcite and crehate @xcite , and across two dimensions: gender and ethnicity. as bias, we refer to systematic discrimination against a disadvantaged group of people @xcite , who will suffer from representational harms because systems tend to fail to recognize their existence @xcite . we think that measuring the uncertainty of models through the lens of annotators with different sociodemographics could help to identify social biases against vulnerable groups. to this end, we exploit the conformal prediction framework to assess the uncertainty and reliability of model predictions. unlike conventional approaches that prioritize accuracy, conformal prediction provides a metric for evaluating the alignment between model outputs and the confidence required for fair decision-making. by identifying disparities, this framework offers a structured approach to understanding biases, ultimately fostering fairness and inclusivity in model design and evaluation. specifically, we formulate two research ques-tions. rq 1. is a models' uncertainty in automatic content moderation a predictor of biases against vulnerable groups to discrimination? rq 2. can the fairness of models be assessed using user representations based on uncertainty? addressing these questions, we show that our approach brings out general patterns of hidden discrimination against non-white people but also shows that some street-level algorithms are expected to be fairer than others. the main contributions of this research are the following: i. we introduce an unsupervised approach that leverages uncertainty to assess the fairness of models' predictions; ii. we provide a benchmarking analysis of 11 nlp systems that exhibit different levels of alignment with annotations provided by annotators belonging to vulnerable groups to discrimination; iii. we demonstrate that representing users through the uncertainty of model predictions is effective to observe the tendency of models to align with specific socio-demographic groups. ## related work unlike structured domains where uncertainty estimation is often a standard consideration, nlp research has traditionally focused on maximizing accuracy-based evaluation metrics, such as f 1 score and log-likelihood, underestimating the computation of model uncertainty. uncertainty measures reached, only recently, visibility in the nlp community @xcite . nlp tasks, indeed, involve inherently ambiguous data, where human label variation introduces significant variability into annotated datasets. and, in such cases, conventional evaluation metrics fail to capture the full extent of model uncertainty, potentially leading to overconfident and unreliable predictions. bias detection. scientific literature has revealed important biases coming from data created and annotated by specific segments of the population, leading to the creation of non-neutral models @xcite and to the reinforcement of social stereotypes @xcite . various techniques have been proposed to reveal biases and models' viewpoints: evaluation of contextualized word embeddings @xcite , specific evaluation frame-works @xcite , questionnaires @xcite , transformer-based recognizers @xcite , special prompts @xcite , and interaction with users @xcite . moreover, recent theoretical frameworks @xcite underline the need to take into account various perspectives about linguistic and pragmatic phenomena. especially the detection of subjective phenomena (i.e., toxic language) proved to be affected by different perceptions that reflect annotators' backgrounds, beliefs, values, and identities @xcite . therefore, a content moderation system should be representative of these different opinions, especially if these opinions come from segments of the population that are actually targets of attacks online @xcite . focusing on toxic language detection, datasets like sbic and crehate with multiple annotations and information about annotators have been proposed and proved to be useful for investigating biases (as we have done in this work), building inclusive @xcite and personalized models @xcite , and providing informative explanations about models' decisions @xcite . confidence and multiple annotations. the most common method to estimate confidence in models is the logit-based method that assesses their uncertainty using token-level probabilities employable to llms @xcite and other models @xcite . @xcite , softmax-based measure of uncertainty has been employed to analyze the level of confidence of models trained on the annotations of specific segments of the population compared with a model trained on majority voting decision, showing that the formers tend to make a decision with less uncertainty than the standard model. similar results are reported by @xcite , where the use of multi-ground truths models, trained on instance-annotator label pairs, improved confidence for samples characterized by substantial annotation disagreements. in this last work, the confidence is computed as the mean class probability for each data's gold label across the epochs. ## methodological framework our methodology relies on conformal prediction, a statistical framework used to quantify the reliability of model predictions by assessing conformity, or how well individual predictions align with a set of labels @xcite . we leverage this theoretical framework to design two metrics for the analysis of bias in pre-trained models with the specific aim to measure their uncertainty against four socio-demographic groups based on the intersection of gender and ethnicity: white men, white women, non-white men, and nonwhite women. ## experimental setup in our experimental setting, we provide two studies that rely on the conformal prediction framework to assess the presence of bias in pre-trained models against four socio-demographic groups based on the intersection between gender and ethnicity: white men, white women, non-white men, and nonwhite women. the first study explores if the adoption of uncertainty can be a predictor of biases against vulnerable groups (rq1) and leverages the uncertainty divergence (section 3.1). the second study assesses models' fairness in user representation (rq2) through the demographic divergence metric (section 3.2). we analyze the uncertainty of 8 fine-tuned lms and 3 llms in the classification of hate speech. we adopt as a benchmark two disaggregated corpora annotated for hate speech, which include information about annotators, such as gender and ethnicity. therefore, we are able to measure the uncertainty of each model against specific communities of people. 2 pre-trained models. in order to account for different generations of models, we considered for our benchmark study a set of transformer-based language models, including both fine-tuned lms and prompted llms in a zero-shot setting. for fine-tuned lms, we based our selection on nlp community adoption metrics: we included all models with at least 1, 000 downloads on huggingface 3 during november 2024. this criterion yielded 8 language models for our experiments, representing a broad spectrum of training methodologies. as a result, we identified 8 lms for our experiment that have been trained with a wide range of approaches: furthermore, we selected 3 open-source llms to replicate the experiment in a zero-shot setting. ## study 1: models uncertainty towards socio-demographic groups in this study we adopt uncertainty divergence (section 3.1) to compare models' performance with their uncertainty against texts labeled by annotators belonging to four socio-demographic groups: white men (w.m.), white women (w.f.), non-white men (¬w.m.), and non-white women (¬w.f.). we compute the f 1 score obtained with each model against the total list of disaggregated labels and the f 1 score against labels of a specific group. this enables ranking the general performance of each model and observing differences between groups. the results in the study shows that uncertainty divergence might be a reliable metric to investigate social biases emerging in classification. the metric does not correlate with the performance of models @xcite . computing the t-test @xcite between f 1 scores and conformity ∆'s show that these two scores are not correlated both in sbic (@xmath0.14) and crehate (@xmath1.11). this suggests that models with a higher performance but a lower conformity might fail in understanding unseen messages outside corpora distribution. in this sense, the higher conformity ∆ assigned to annotations provided by non-white people can be interpreted as a predictor of a potential systematic misalignment between street-level algorithms decisions and women perception of hate speech. therefore, conformity might be not only used as a metric of fairness but as a guiding principle for selecting for content moderation models that are able to see through the lens of vulnerable minorities. ## study 2: user representation: clustering annotators according to their uncertainty this study leverages demographic divergence to represent users through models' uncertainty in text classification. the annotators were grouped into four distinct clusters for each model m as described in section 3.2. the number of clusters was set to 4 to align with the examined sociodemographic groups (w.m., w.f., ¬w.m., ¬w.f.). clustering was performed on the annotator vectors v m a , ∀@xmath2, which represent the distribution of uncertainty for each annotator, using k-means. figure 1 shows an example of results based on hate-bert uncertainty scores over the crehate dataset. bar plots represent the demographic distribution of annotators in each cluster, and line plots represent the uncertainty related to each socio-demographic group. all the clusters are available in appendix b. at this stage, for each of the 11 models considered, we have two clusterings (one for the cre-hate dataset and one for the sbic dataset), each consisting of 4 clusters. our goal is twofold: on the one hand, to quantify the distribution of demographic characteristics of the annotators within the clusters, and on the other, to evaluate uncertainty divergence within individual clusters across the models. for a model m and the four identified annotator clusters, we calculate the average uncertainty across demographic classes for each cluster by leveraging uncertainty divergence metrics. if the average uncertainty varies across the different clusters, it indicates that there are groups of annotators, for which the model m exhibits lower fairness compared to others. figure 2 illustrates the ranking of models based on uncertainty divergence and demographic divergence. uncertainty divergence and demographic divergence are specifically used to quantify the fairness of models @xcite . by examining the figure, we can see that the llms exhibit higher uncertainty divergence for both datasets. therefore, despite achieving optimal results in terms of f 1 , the higher uncertainty compared to the lms indicates that there are groups of annotators for which llms exhibit lower fairness. in contrast, demographic divergence helps us to understand whether this uncertainty is distributed equitably across the clusters, and hence, across the demographic classes. llms perform better in the ranking, particularly mistral-7b. consistent with tables 1 and 2 , this llm maintains fairness across the considered dimensions (gender and ethnicity). on the other hand, olmo-7b presents negative demographic divergence values, indicating higher uncertainty, which is not evenly distributed across the demographic classes. we want to highlight the behavior of muril, which exhibits the lowest uncertainty among all models for both datasets but presents the highest demographic divergence. in appendix b, figure p shows that clusters with female or non-white female annotators exhibit higher uncertainty, and the distributions between clusters are significantly different. ## discussion our research shows that uncertainty and the conformal prediction framework are a powerful approach for the analysis of models' fairness in automatic content moderation. uncertainty is effective for the observation of the alignment of models' behavior with the sensibility of specific groups of annotators. uncertainty is also a powerful way to understand how single annotators are represented and grouped, and to what extent models perpetuate social biases during this process. from the benchmark analysis of pre-trained models, both general patterns and model-specific behaviors emerge. the vast majority of models show the lowest uncertainty in predicting contents annotated by men and the highest uncertainty for the prediction of contents annotated by non-white people. this implies that in the automatic content moderation settings, in which the model performance might suffer a high degradation due to temporal semantic shifts, the risk of misalignment between the model's predictions and human annotations is higher for non-white people. an explanation of such a pattern might be in the long-term impact of the pretraining process. models trained on data that poorly represent non-white people learn perspectives of the world that are not easy to be removed. in this sense, the more nuanced results in terms of performance might be interpreted as an optimization over specific benchmark corpora, while uncertainty could be the blueprint of pretraining biases. despite the presence of common behaviors, choosing one pre-trained model over another matters. the average uncertainty of these models significantly varies without correlating with performance (section 5), showing that the two ways of measuring them (f 1 score and conformity ∆) grasp different aspects of the same problem. in this sense mistral represents the best trade-off between performance and uncertainty, despite the lack of information about which data has been used to train it @xcite . however, it is not possible to draw conclusions from the comparison of fine-tuning and zero-shot approaches, since olmo and bloom exhibit a higher demographic divergence (section 6) within their clusters: a predictor of bias against vulnerable groups to discrimination. corpora themselves appear to be a factor in stressing the uncertainty of models and represent a significant limitation for fairness studies. model predictions on sbic systematically suffer higher demographic divergence (appendix b) than predictions on crehate. the very different composition of annotators (section 4) might play a role in this, as well as the different degrees of subjectivity emerging from raw annotations (4.9% vs 9.7%), and the average number of annotations per message (3.2 vs 26.9). furthermore, the inter-annotator agreement between annotators belonging to different demographic classes is generally higher within the same class, while agreement between different classes tends to vary (appendix c). despite these differences, the two corpora share the same limitation: binarism in the annotators' selection process. non-binary people were almost not involved in the annotation process, hindering most insightful analyses that go beyond the traditional intersection of race and gender. resources developed for fairness should be more effective in representing marginalized and invisible groups of people. ## conclusion and future work in this paper, we presented a novel approach to assess the fairness of models through their uncertainty. we introduced metrics for measuring the impact of uncertainty against socio-demographic groups. in particular, we leveraged our unsupervised approach based on conformal prediction to benchmark 11 street-level algorithms on sbic and crehate datasets: 8 lms fine-tuned for hate speech detection and 3 llms instantiated through a prompt-based method. the results show that measuring models' uncertainty unfolds systematic and hidden biases against non-white people, which do not emerge from performance-based metrics, such as the f 1 score @xcite . moreover, we generated vector representations of annotators based on uncertainty scores emerging from models' predictions and used them to cluster annotators. the socio-demographic composition of the resulting clusters significantly varies between models, which show different degrees of fairness against women and non-white people @xcite . future work goes in two directions. we will test the impact of considering uncertainty during fine-tuning and active learning (e.g., through reinforcement learning approach) to reduce bias in model prediction. we will explore the transferability of our methodology on contiguous tasks to hate speech detection and to other perspectivist corpora.