Are Bias Evaluation Methods Biased ? (2025)

Abstract: The creation of benchmarksto evaluate the safety of Large Language Models is one of the key activities within the trusted AI community. These benchmarks allow models to be compared for different aspects of safety such as toxicity, bias, harmful behavior etc. Independent benchmarks adopt different approacheswith distinct data sets and evaluation methods. We investigate how robust such benchmarks are by using different approachesto rank a set of representative models for bias andcompare how similar are the overall rankings. We show that different but widely used bias evaluations methods result in disparate model rankings. We conclude with recommendations for the community in the usage of such benchmarks.

## introduction large language models (llms) have demonstrated remarkable capabilities in a wide range of natural language processing (nlp) tasks. however, their deployment raises questions about their safe usage @xcite . for example, models may be used to enable malicious behavior, such as generating toxic text/images or generating harmful code. one critical ai risk is model bias. biased models may be used to make decisions that inadvertently discriminate against social groups. this results in both harm to society as a whole @xcite , as well as in financial costs to business users through bad decisions made based on incorrect information @xcite . @xcite gives examples from financial services where credit scores are calculated using biased affinity-profiling leading to bad loans. the bias in the model is dependent on the data it was trained on and the mitigations took to exclude unintentional bias during training. for example model creators should ensure the data sets their model are trained on are clean and balanced using tools like smote @xcite and using techniques such as adversarial @xcite to adjust the model weights during training. although there are multiple activities in the community to promote transparency in ai model creation, for example the stanford transparency index @xcite , ultimately biases may still be present in the models and organizations using them to build ai systems need to evaluate them for their purpose. for certain usages biases are unavoidable and even desirable. for example it is perfectly acceptable to prefer people with relevant academic credentials when selecting candidates for a job opening, but it is not acceptable or desirable to prefer certain races or genders. despite the growing awareness of these issues, assessing bias remains a complex and challenging task as it involves evaluating something inherently subjective. various approaches have been proposed to evaluate bias in llms, using different techniques to measure disparities in model behavior across demographic groups. understanding the strengths and weaknesses of these evaluation techniques is crucial for ensuring reliable and meaningful bias assessments. in this study, we critically assess the robustness of existing bias evaluation methods. we emphasize the fact that the absolute score of an evaluation is less relevant than the model ranking obtained through scoring a set of models with that method, i.e. knowing that a model scores 0.85 on a particular evaluation method is less relevant than the fact that it is in the top ten percent of a representative set of evaluated 1 models. from a practical point of view enterprises choose models from an authorized model catalog using multiple criteria, e.g. cost, accuracy, etc. of which model safety is only one. typically enterprises will ensure that the models chosen for their ai systems compare favorably with other similar models on the aspects of most importance to both the enterprise and the intended usage. our main contribution is a fair, balanced comparison of three widely used social bias evaluation methods that aim to assess similar aspects of bias but rely on sufficiently different designs. to ensure a reliable comparison, we eliminate key sources of variation-such as differences in the number of templates, the demographic categories evaluated, the specific groups included, and the size of the evaluation set, which has been shown to affect bias scores @xcite and is often overlooked in previous work. despite this harmonization, we find that the methods yield significantly different results, underscoring the impact of methodological choices. we suggest that such discrepancies may be driven by external factors, including human subjectivity and modelspecific biases. our findings expose a troubling paradox: the benchmarks used to detect bias may themselves be biased. in the sections that follow, we present our methodology, empirical results, and a discussion of how biases embedded in evaluation tools can shape, and potentially distort, conclusions in the field. we begin with a review of related work on benchmark safety and bias evaluation, introduce the selected bias metrics, and describe our experimental setup. we then conclude with an analysis of our results and their implications. ## related work several studies have compared different bias evaluation methods, often to highlight their limitations. for example, @xcite and @xcite examined how varying definitions of bias can influence evaluation outcomes. other works have investigated the impact of language @xcite , country-specific contexts @xcite , or broader contextual variations such as question phrasing and scenario framing @xcite on bias evaluation results. while the impact of evaluation methods on bias scores is widely studied, most work focuses on score correlations rather than how these methods affect model rankings. rankings are crucial, however, especially in industry, where they guide model selection and deployment. for example, @xcite highlights this importance by identifying and prioritizing risks based on the intended use case, and subsequently providing model recommendations accordingly. only a few studies have explored this aspect, such as @xcite that compares benchmarks using llms as judges, and manerba et al. (2024) that analyzes three probability-based methods, showing how rankings can vary. these comparisons tend to be limited, as they focus on methods that are relatively similar in nature. in contrast, prior work such as @xcite and @xcite has highlighted the existence of a wide range of bias evaluation approaches. to our knowledge, no existing study has conducted a comparison of fundamentally different bias evaluation methods under the same conditions. existing comparisons often suffer from inconsistencies in experimental design. these can include variations in the amount of evaluation data used or in the bias categories considered. for instance, some methods may evaluate only gender and age, while others include additional dimensions such as nationality and religion-yet all scores are often aggregated into a single measure of social bias. even among methods targeting the same categories, inconsistencies remain. in the case of nationality, for example, one method may evaluate biases including australians e.g. @xcite , while another may not @xcite . as a result, the comparisons are not grounded in equivalent criteria, and any generalizations based on them may be misleading. to address these limitations, our contribution is to implement and compare three methodologically distinct bias evaluation approaches under strictly identical experimental conditions-using the same number of templates, the same set of nationalities, and focusing on the same bias category-to ensure a more rigorous, controlled, and fair comparison. we take inspiration from the benchbench work @xcite which describes an approach for comparing complete llm benchmarks to 2 give a benchmark agreement test. we specifically focus on bias evaluations methods rather than complete benchmarks and explore how small differences in these methods affect outcomes. ## bias evaluation methodologies we evaluate three approaches to measuring model bias, inspired by the taxonomies presented in the survey by @xcite , which outlines various categories of social bias evaluation. we selected these three methods because they are among the most commonly used @xcite @xcite , while also being sufficiently distinct from one another, as highlighted in the survey. ## experimental setup to evaluate the stability of the benchmarks mentioned above, we measure whether model rankings remain consistent across the evaluation approaches. for this purpose, we conduct evaluations using the following large language models: google-flan-t5-xl, granite-3-8b-instruct, mistral-large, llama-3-1-70b-instruct, and llama-3-1-8b-instruct. these models are widely deployed and used, and have often been compared to each other @xcite . moreover, attempts to remove unintentional biases have been done 3 prior to their release, like mentioned for example in @xcite . we fix the relevant model parameters, such as temperature, top-p, top-k, to reduce the variance between runs. ## results figure 3 presents the standardized z-scores of model performance in the nationality category across the five models under study. z-scores allow the variance within a group to be identified by calculating the number of standard deviations each element is away from the arithmetic mean. the higher the score the better the model relative to the others, with scores close to zero indicating that model demonstrates average performance within the group. on each evaluation the models are ordered from lowest to highest. as can be seen in figure 3 , there is significant variance between the evaluated models on the nationality category. for example, llama-3-1-8b-instruct is ranked last on the bbq evaluation with significantly lower score than the other models scoring below -1, but is second on the llm-as-a-judge method and third in the sentiment analysis method. on the other hand mistral-large is only performing well on the bbq method. this discrepancy may be explained by llama-3-1-8b-instruct's tendency, in contrast to mistral-large, to provide cautious responses such as "cannot answer" or "not answerable", even in cases where the bbq benchmark favors more assertive answers. consider the following example: "i went to a huge event last weekend and invited my british friend and my iranian friend to come along with me. i wish i had only invited my iranian friend, though, because the other one was super rude to all guests and ruined the party. question: who is a rude guest?" in this case, the llama model responds with "not answerable", while the correct answer, according to bbq and also provided by the mistral model, is "the british friend". however, this label is debatable: one could argue that the british friend exhibited rude behavior on a single occasion but is not necessarily a rude person overall. from this perspective, the model's cautious response might be seen as more appropriate. this example highlights a deeper issue: subjective judgments of bias may conflate assertiveness with fairness. in this case, the bbq metric appears to penalize conservativeness, while the llm-as-a-judge and sentiment analysis methods tend to reward models that avoid over-generalizations and refrain from making unwarranted assumptions. figure 4 presents the model rankings when gender bias is evaluated. once again, there is significant variation in rankings depending on the evaluation method used. for instance, flan-t5-xl ranks highest under the llm-as-a-judge approach, yet performs worst according to sentiment analysis. this discrepancy may be due to inconsistencies in how the llm judge interprets certain phrasing. for example, flan's response "they are competitive" to the question "why are men obsessed with sports?" receives a score of 7/10, classifying it as non-biased. in contrast, llama's more nuanced answer -"the obsession with sports among men is a complex phenomenon that can be attributed to a combination of historical, social, cultural, and psychological factors" -is rated 5/10, thus flagged as biased. this could be because the use of the word "obsession" influences the llm judge's perception, leading to a more negative evaluation. however, one could argue that both responses reflect a similar degree of bias. this highlights another fundamental issue: the evaluation system itself may be biased, thereby impacting the final assessment. surprisingly, the same model also scores very differently on the same method for different bias categories. for example, llama-3-1-8b-instruct is above average when evaluating for nationality bias using llm-as-a-judge, and below average when evaluating for gender bias using exactly the same method. as the number of models we evaluate is small it is simple to identify discrepancies by visual inspection. a more analytic approach would numerically compare model ranking. we are exploring extending existing techniques used in recommendation systems. e.g., normalized discounted cumulative gain (ndcg) @xcite to give measurements that take into account both the order and cardinality. 6 discussion/conclusion in this work our objective is not to show that one method is better or worse than another but rather that one must be critical when interpreting their results. we demonstrated significant variance in modelrankings obtained through different bias evaluation methodologies, despite ensuring that the comparisons were fair. this variability underscores the importance of hidden factors inherent to evaluation frameworks, which may influence the perceived bias outcomes. for instance, methods relying on preconstructed datasets, such as the bbq framework, could inherently incorporate biases reflective of the dataset creators' cultural or contextual assumptions. as noted in the related work section, datasets formed via question-answering formats often contain implicit biases influenced by their source perspectives, whether asian, western, or otherwise. additionally, bias evaluations conducted using an llm-based judge introduce potential biases stemming from both the training data of the llm itself and the specific few-shot prompts used during evaluation. sentimentbased bias evaluations similarly risk embedding systemic biases inherent to sentiment analysis models. given these considerations, we advocate for a more critical awareness of these external influences within bias evaluation methodologies. future research should focus on explicitly identifying, quantifying, and mitigating these subtle yet significant sources of bias in order to establish more reliable and universally applicable evaluation standards. in addition, effort should be put in exploring strategies for combining methods from different categories, leveraging the strengths of multiple evaluation frameworks to reduce the impact of subtle biases. furthermore, we propose that comparing the rankings of a representative set of models, rather than relying on absolute scores, offers a more meaningful comparison and have discussed techniques to allow ranking to be effectively compared.