Beyond Performance: Quantifying and Mitigating Label Bias in LLM s (2024)

Abstract: Large language models (LLMs) have shown remarkable adaptability to diverse tasks, by leveraging context prompts containing instructions, or minimal input-output examples. However, recent work revealed they also exhibit *label bias*—an undesirable preference toward predicting certain answers over others. Still, detecting and measuring this bias reliably and at scale has remained relatively unexplored. In this study, we evaluate different approaches to quantifying label bias in a model’s predictions, conducting a comprehensive investigation across 279 classification tasks and ten LLMs. Our investigation reveals substantial label bias in models both before and after debiasing attempts, as well as highlights the importance of outcomes-based evaluation metrics, which were not previously used in this regard. We further propose a novel label bias calibration method tailored for few-shot prompting, which outperforms recent calibration approaches for both improving performance and mitigating label bias. Our results emphasize that label bias in the predictions of LLMs remains a barrier to their reliability.

## introduction large language models (llms) have demonstrated impressive abilities in adapting to new tasks when conditioned on a context prompt, containing tasksolving instructions @xcite or few examples of input-output pairs @xcite . still, recent work has shown that predictions of llms exhibit label bias-a strong, undesirable preference towards predicting certain answers over others @xcite @xcite , see fig. 1 ). such preferences were shown to be affected by the choice and order of in-context demonstrations @xcite , the model's pretraining data @xcite , or figure 1 : llms exhibit label bias-a tendency to output a given label regardless of the context and input (in this example, 'yes' over 'no'). in this work we evaluate llm label bias across ten llms and 279 classification tasks, showing label bias is a major problem in llms. despite these efforts, label bias evaluation relies on performance metrics such as accuracy, rather than metrics designed to directly quantify the bias. in doing so, we might inadvertently overlook crucial aspects of model behavior. indeed, although a given method could effectively improve performance, substantial bias might still persist in the model's predictions-deeming the method insufficient and the model unreliable. alternatively, performance could remain relatively unchanged, but with the bias mostly removed. in this work, we take a step towards a more comprehensive understanding of the extent of label bias in llms and the effects of mitigation approaches. using metrics to directly measure the label bias in model predictions, which we derive from previous work on fairness and label bias esti-mation, we evaluate ten llms on 279 diverse classification and multiple-choice tasks from super-naturalinstructions @xcite . we examine both performance and bias along axes such as scale and number of in-context demonstrations. we also evaluate the impact of label bias mitigation methods, such as calibration and fewshot lora fine-tuning @xcite . our investigation reveals substantial label bias in the predictions of llms across all evaluated settings, indicating that raw llm output scores often represent simple, heuristic solutions. while increasing model size, providing in-context demonstrations, and instruction-tuning all contribute to reducing bias, ample bias persists, even after applying mitigation methods. surprisingly, these results also hold for tasks where the labels are all semantically equivalent (e.g., in multi-choice question answering). further, although the examined calibration methods can reduce bias and improve performance, we also find cases where they negatively impact both bias and overall performance. motivated by these findings, we propose a novel calibration method for few-shot prompting that more accurately estimates a model's label bias, using only its predictions on the in-context demonstrations. compared to existing llm bias calibration methods, our method improves performance while also removing considerably more bias. our findings highlight the necessity of considering and measuring biases in the predictions of llms when evaluating their performance. moreover, adjusting models to their tasks through more accurate and effective estimation of biases holds promise for improving the reliability of llms and their applications. ## llm label bias our objective is to broaden the understanding of label bias in llms and the effectiveness of mitigation strategies, focusing on classification tasks. in this section, we define metrics designed to quantify bias in model predictions, providing a nuanced examination of label bias that extends beyond traditional performance metrics. we describe the setting of label bias in in-context learning ( §2.1), briefly outline methods to mitigate it ( §2.2), and finally review approaches to evaluate label bias as well as define the metrics we use in this work ( §2.3). ## mitigating label bias by calibrating on demonstrations motivated by the challenges of existing calibration approaches on instruction-tuned models ( §4.3), we aim to develop an effective calibration method for such scenarios. we hypothesize a possible cause for such difficulties is that the inputs used for calibration in cc and dc are very distinct from the more curated, high-quality inputs models observe during instruction-tuning @xcite . 5 seeking to use more naturally-occurring inputs, and to avoid any reliance on additional held-out examples, we propose to calibrate models using the in-context examples readily available in few-shot prompts. we therefore need to obtain the model's output probabilities on these inputs to estimate its bias. however, as these examples appear alongside their labels in the context provided to the model, it could simply copy the correct answer from the prompt, leading to unreliable bias estimates. we introduce a simple method to alleviate this concern. leave-one-out calibration (looc) our goal is to estimate the model's average output probabilities p at test-time by using the k demonstrations rpx 1 , y 1 q, . . . , px k , y k qs provided in the context c, and then use it for calibration. drawing from leave-one-out cross-validation, when evaluating the model on the i-th demonstration's input x i , we prompt it with an edited context c ´i comprised of the original context c after removing the current demonstration px i , y i q, resulting in k ´1 demonstrations. 6 we thus obtain k output probabilities: to reliably estimate p, we further need to account for the demonstrations' labels y i : for imbalanced choices of demonstrations (e.g., class imbalance), using the average of p i 's could lead to an underestimation of the probability assigned to infrequent labels. we therefore compute the average output probabilities p by taking into account the labels y i , as we do for computing biasscore ( §2.3): we first average p i 's associated with the same label ℓ, d ℓ " tp i | y i " ℓu, and then set p as the mean of these intra-label averages: finally, we use p to compute calibration parameters and score new examples using the same methodology as @xcite . we refer to our method as leave-one-out calibration (looc). ## analysis we study the effect of different factors on the extent of label bias in model predictions: the semantic meaning of the task labels ( §6.1), the level of label imbalance in the demonstrations ( §6.2), and the choice of demonstrations ( §6.3). ## conclusion the label bias of llms severely hinders their reliability. we considered different approaches for quantifying this bias. through extensive experiments with ten llms across 279 classification tasks, we found that substantial amounts of label bias exist in llms. moreover, we showed this bias persists as llms increase in scale, are instructiontuned, are provided in-context examples, and even when they are calibrated against such bias. we proposed a novel calibration method, which outperforms existing calibration approaches and reduces label bias dramatically. our results highlight the need to better estimate and mitigate llm biases.