Rethinking Prompt-based Debiasing in Large Language Model (2025)

Abstract: Investigating bias in large language models (LLMs) is crucial for developing trustworthy AI. While prompt-based through prompt engineering is common, its effectiveness relies on the assumption that models inherently understand biases. Our study systematically analyzed this assumption using the BBQ and StereoSet benchmarks on both open-source models as well as commercial GPT model. Experimental results indicate that prompt-based is often superficial; for instance, the Llama2-7B-Chat model misclassified over 90% of unbiased content as biased, despite achieving high accuracy in identifying bias issues on the BBQ dataset. Additionally, specific evaluation and question settings in bias benchmarks often lead LLMs to choose “evasive answers”, disregarding the core of the question and the relevance of the response to the context. Moreover, the apparent success of previous methods may stem from flawed evaluation metrics. Our research highlights a potential “false prosperity” in prompt-base efforts and emphasizes the need to rethink bias evaluation metrics to ensure truly trustworthy AI. We will release our data and code upon acceptance.

## introduction as large language models (llms) advance, addressing their inherent biases is critical for responsible ai, especially in high-stakes domains like education, criminal justice, and media @xcite @xcite @xcite . prompt-based methods have become a popular debiasing approach @xcite , widely adopted for current llms due to their accessibility and perceived effectiveness. however, the efficacy of these prompt-based techniques often rests on an implicit assumption: corresponding authors. to systematically investigate this, we propose a two-stage analytical framework, as shown in figure 1 . first, we scrutinize the fundamental capability of llms to accurately identify and understand various manifestations of bias. second, we evaluate the practical effectiveness and reliability of promptbased self-debiasing methods when applied to these models. this framework guides our inquiry into whether llms' responses to debiasing prompts reflect genuine comprehension or a more superficial pattern matching. our examination of open-source and commercial models supports our hypothesis, revealing llm limitations in bias identification. for instance, llama2-7b-chat, while detecting explicit bias, misidentifies bias in 90% of unbiased scenarios, suggesting llm alignment may be more superficial than thought @xcite . furthermore, our experiments reveal two indicators of superficial self-debiasing: inconsistent results from prompt-based methods across different settings, and llms frequently resorting to "evasive responses" that avoid core issues when prompted on social bias. these findings collectively challenge the presumed effectiveness of current prompt-based debiasing approaches, revealing them to be potentially superficial and unstable. critically, we also find that evaluation metrics are not innocent bystanders; the metrics commonly used to evaluate debiasing effectiveness often fail to capture the prevalence of prompt-based debiasing (c) unknown ambiguous contexts disambiguated contexts (context 1) (context 1+2) a b c w/ unknown w/o unknown (c) unknown (a) nancy (a) nancy a b c a b a b reject answer reject answer who has a mental health diagnosis? ## analytical methodology although evaluation metrics from various bias benchmarks seem to have shown good results in prompt-based debiasing approaches, several critical studies have raised important concerns about these methods. @xcite conducted a thorough analysis of fairness benchmark datasets, revealing significant limitations in how these datasets conceptualize stereotyping. their work, along with other analytical studies @xcite , challenges the fundamental assumption that llms can autonomously correct their biases through prompting alone, without more substantial interventions. motivated by these concerns, we propose a systematic evaluation framework to investigate whether llms truly understand and effectively address bias, or if they merely exhibit surface-level pattern matching. this distinction is crucial as it directly impacts the reliability and effectiveness of prompt-based debiasing methods. our analysis framework consists of two main components: ## conclusions in this paper, we examined the efficacy and implications of prompt-based debiasing methods in llms using three widely recognized bias benchmarks. our findings reveal a complex landscape in which attempts to mitigate bias can often be fragile and harmful to other capabilities. the entire process behind self-debiasing may employ evasive tactics, complicating the straightforward interpretation of current debiasing metrics. there is a need for more nuanced evaluation metrics and techniques that balance bias mitigation with the preservation of llm performance across diverse contexts.