Social Bias Evaluation for Large Language Models Requires Prompt Variations (2025)

Abstract: Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mitigate these biases accurately. Previous studies use downstream tasks to examine the degree of social biases for evaluation and mitigation. While the output of LLMs highly depends on prompts, prior works evaluating and mitigating bias have often relied on a limited variety of prompts. In this paper, we investigate the sensitivity of LLMs when changing prompt variations (task instruction, few-shot examples, debias-prompt) by analyzing task performance and social bias of LLMs. Our experimental results reveal that LLM rankings fluctuate across prompts for both task performance and social bias. We also confirmed that the impact of format changes can differ for each bias category. Performance improvement from prompt settings may not result in reduced bias. Moreover, the ambiguity of instances is a common factor in LLM sensitivity to prompts across advanced LLMs. We recommend using diverse prompts, as in this study, to compare the effects of prompts on social bias in LLMs.

## introduction while llms have high performance, they also have unfair and severe social biases, which can harm specific groups @xcite @xcite . in response to these concerns, many prior studies have tackled to assess and mitigate social bias in llms. social biases in llms are often evaluated using the llms' predictions in downstream tasks such as question answering @xcite , natural language inference @xcite , commonsense reasoning @xcite , sentence completion @xcite . recent llm developers task instruction debias prompt few-shot examples bias score task performance insert various categories gender race religion disabiltity gap ! figure 1 : prompt variations on bias evaluation: this example shows prompt variations on bias evaluation using downstream task (1) task instruction, (2) fewshot examples, and (3) debias-prompt. these variation factors can affect the scores. the instance was sampled from the bbq dataset @xcite . as for mitigation of social bias, various methods have also been proposed, such as counterfactual data augmentation @xcite , decode intervention @xcite , and text intervention @xcite . although llms should be robust to input variations, challenges remain in the evaluation due to the sensitivity regarding the prompts @xcite @xcite . previous studies have highlighted that llms have the sensitivity to task instruction @xcite @xcite , and verification with multiple context ambiguous we couldn't start the board meeting at 9am today because a man and a woman were late. disambiguated (stereotype) {ambiguous context} + we had to wait for the man because they were running the meeting, and the woman was responsible for taking all the notes. disambiguated (anti-stereotype) {ambiguous context} + we had to wait for the woman because they were running the meeting, and the man was responsible for taking all the notes. ## bias evaluation on llms using the downstream task this paper focuses on bias evaluation in the form of multiple-choice questions (mcqs), which are commonly used for assessing llms' ability @xcite . in the mcqs setting, the llms are required to choose the most suitable answer from the candidate answers. to comprehensively evaluate llms' sensitivity, we prepared three prompt variation factors. ## experiments in this section, we investigated the sensitivity of llms in the zero-shot and few-shot settings. we looked into whether the few-shot setting can mitigate llms' sensitivity and how it affects task performance and bias scores compared to the zero-shot setting. to quantify sensitivity, we calculate the sensitivity-gap @xcite , which is the difference between the maximum and minimum llms' scores, such as task performance or bias scores, as follows. ## analysis to investigate the prompt sensitivity of llms in more detail, we analyzed our results from four aspects: correlations across different prompt formats ( §4.1), correlations across different bias categories ( §4.2), correlations among different metrics ( §4.3), and the instance-level sensitivity ( §4.4). before our analyses, we define a matrix of scores s (c,m) ∈ r l×f , where l and f denote the numbers of llms and prompt formats, respectively (@xmath0. c represents one of bias categories: {gender, race, religion, disability}, and m represents one of metrics: { acc a , acc d , diff-bias a , diff-bias d }. an element s (c,m) i,j ## debias-prompt we examined how debias-prompts affect evaluation metrics. debiasing via prompting is a promising method to mitigate social bias because it does not require additional model training and can only work with additional text input. we call this kind of prompt debias-prompt. although prior work verified the effectiveness of debias-prompt on bias evaluation dataset @xcite @xcite , these studies only verified limited prompts or models. therefore, comparing the effectiveness of debias-prompts is important. setting we investigated the effectiveness of debias-prompts across formats and models in the same few-shot setting as in section 3. we created 12 debias-prompts, such as note that the sentence does not rely on stereotypes, using the template in terms of three perspectives (level, style, and negation) based on the prior work (described in appendix b). we inserted the debias-prompt at the beginning of the prompt. for simplicity, we focus on maximum and minimum values across different debias-prompts on average over 10 prompt formats. result analysis we also examined the effectiveness of debias-prompts across different prompt formats. we calculate kendall's τ coefficient to measure the ranking correlation between format pairs as in §4.1 regarding 12 debias-prompts. bias evaluation in downstream tasks existing studies investigate how to quantify social biases in downstream tasks such as text generation @xcite @xcite , coreference resolution @xcite , and machine translation @xcite . as for question answering, @xcite developed unqover datasets by using ambiguous questions to assess model biases and ambiguity was followed by later research @xcite . prior work using the downstream task for llms mainly focuses on bias evaluation score on llms; in comparison, our work mainly focuses on llms sensitivity in bias evaluation. we conducted an exhaustive survey, a research style widely adopted and recognized as valuable in the study of social bias @xcite . robustness of llms our study is related to the robustness of llms @xcite @xcite @xcite as for mcqs, surface change can affect performance, such as choice order @xcite , prompt format @xcite , task description @xcite , case description @xcite , and calculation of choice selection @xcite . in this work, we investigated the robustness of task performance and social bias of llms simultaneously from multiple perspectives. ## conclusion this study showed that llms are highly sensitive to prompt variation (task instruction, few-shot examples, and debias-prompt) in both task performance and social bias. the sensitivity may lead to fluctuations in the ranking of llms. bias trends under prompt variations can differ for each bias category we confirmed that llms only have weak correlations between task performance and social bias caused by the prompt variations. our analysis indicated that the ambiguity of instances is a common factor in llm sensitivity to prompts across advanced llms. our findings shed light on the bias evaluation of llms derived from their sensitivity. we recommend using prompt variations, as in this study, to compare the effects of prompts on social bias in llms. in future work, we will expand our investigation to other tasks.