Do Prevalent Bias Metrics Capture Allocational Harms from LLM s? (2025)

Abstract: Allocational harms occur when resources or opportunities are unfairly withheld from specific groups. Many proposed bias measures ignore the discrepancy between predictions, which are what the proposed methods consider, and decisions that are made as a result of those predictions. Our work examines the reliability of current bias metrics in assessing allocational harms arising from predictions of large language models (LLMs). We evaluate their predictive validity and utility for model selection across ten LLMs and two allocation tasks. Our results reveal that commonly-used bias metrics based on average performance gap and distribution distance fail to reliably capture group disparities in allocation outcomes. Our work highlights the need to account for how model predictions are used in decisions, in particular in contexts where they are influenced by how limited resources are allocated.

## introduction the rise of large language models (llms) has raised concerns about potential harms in highstakes decisions, such as lending @xcite , hiring @xcite , and healthcare triage @xcite . recent orders in europe @xcite and the u.s. @xcite have mandated audits to address ai risks including bias but left it unclear how to conduct effective audits. several works have conducted bias audits for llms in critical decision-making @xcite @xcite . yet, they focus on the predictions models make, without considering how those predictions would be used to make decisions. even when predictions appear to be unbiased, actual harms can arise from how they are used to make @xcite @xcite . as shown by @xcite , evaluating models in isolation is insufficient to assert fairness without considering the context in which they will be deployed. allocational harms arise if certain groups of people are deprived of access to resources or opportunities @xcite . in settings where resources are limited and a model is used to prioritize options, there is a gap between predictions and decisions. it is unclear whether prevailing metrics, which measure bias in prediction outcomes, are sufficient to measure bias risks in applications where predictions are used for resource allocation. contributions. to assess the potential harms of using llms for decision-making, we evaluate how well commmon bias metrics predict actual disparities in allocation outcomes. these metrics typically rely on average performance and distribution differences. we conduct this evaluation across ten llms on two allocation tasks (section 3). our findings demonstrate that bias metrics based on predictions may not reliably reflect true disparities in outcomes (section 4.1). in addition, these metrics may sometimes identify models with greater disparities as less biased and exhibit inconsistent predictive abilities across different groups (section 4.2). as a more reliable alternative, we propose the rankbiserial correlation, which demonstrates a strong correlation with actual allocation disparities. ## background algorithmic bias is commonly described as "skew that produces a type of harm" towards certain groups of people @xcite . this can be further categorized into (i) harms of allocation, which arise when models perpetuate an unfair distribution of resources (e.g., healthcare) or opportunities (e.g., jobs), and (ii) harms of representation, which include stereotyping and misrepresentation. ## method we consider the allocation task as a top-k ranking problem @xcite , where a fixed quota of @xmath0. the goal is to determine a set of "best" candidates, with no particular emphasis on the relative order. we follow the llm ranking method of @xcite and rank the candidates in descending order of their prediction scores. ## results this section shows results comparing bias metrics and allocation gaps in candidate selection outcomes based on llm predictions. we first present the overall predictive validity, then the utility for model selection and informing bias risks. ## discussion our findings reveal that common bias metrics for evaluating llms do not capture allocational harm. while final decisions may depend on human decision-makers or other external factors, a reliable measurement is crucial for estimating the potential risks of a model. in fact, in settings of unfamiliar domains and objective tasks, humans tend to rely more on model predictions @xcite @xcite . @xcite have shown that algorithmic risk assessments not only alter human decisions but exacerbate racial disparities. the goal of an audit is to determine if it is acceptable to deploy a model. although audits will always be imperfect since they require making predictions about how the model will behave on future data, it is essential that we develop methods for auditing models that reliably measure potential harms in the way models will be used in deployment. our results demonstrate that metrics too far removed from how a model will be used may fail to adequately measure how well the model will perform as deployed. -1 -0.5 0 0.5 -0.4 -0.2 0 0.2 -1 -0.5 0 0.5 -0.2 0 0.2 -0.5 0 0.5 1 -0.5 0 0.5 -1 -0.5 0 0.5 1 -0.5 0 0.5 δ rb δdp δeo 0 0.1 0.2 0.3 0 0.1 0.2 0.3 0.4 0 0.5 1 0 0.1 0.2 0.3 0.4 0 0.05 0.1 0 0.2 0.4 0.6 0 0.5 1 0 0.2 0.4 0.6 jsd emd |δdp| |δeo| figure 8: bias metrics (x-axis) and allocation gaps (y-axis) for essay grading, with quota @xmath1. b.2 metric utility -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 δdp δeo pointwise δ jsd emd rb pearson correlation (r) figure 9: bias metric and allocation gap correlation by group in essay grading with @xmath2. 1 2 3 4 5 6 7 8 9 10 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 0.4 0.6 0.8 1 δ jsd rb rank position n rank position n avg ndcg@n resume screening (@xmath3: average ndcg@n in ranking model fairness, comparing to ideal rankings based on ∆eo. system you are an essay grading assistant. your task is to assess an opinion essay on @xmath4. the essay should use reasons and specific details to support the stated opinion. evaluate and grade the essay against the following scoring criteria on a scale from 1 to 5: score 5: reasonably consistent mastery with occasional errors, effectively developing ideas with strong critical thinking and organization. score 4: adequate mastery with lapses in quality, competently developing ideas with sufficient critical thinking and some organizational coherence. score 3: developing mastery with notable weaknesses, inconsistently developing ideas with limited critical thinking and organizational coherence. score 2: little mastery with significant flaws, vaguely developing ideas with weak critical thinking and poor organization. score 1: very little or no mastery, failing to develop viable ideas with severe disorganization and pervasive errors user essay: <@xmath5.