Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement (2024)

Abstract: Recent studies show that large language models (LLMs) improve their performance through self-feedback on certain tasks while degrade on others. We discovered that such a contrary is due to LLM’s bias in evaluating their own output. In this paper, we formally define LLM’s self-bias – the tendency to favor its own generation – using two statistics. We analyze six LLMs (GPT-4, GPT-3.5, Gemini, LLaMA2, Mixtral and DeepSeek) on translation, constrained text generation, and mathematical reasoning tasks. We find that self-bias is prevalent in all examined LLMs across multiple languages and tasks. Our analysis reveals that while the self-refine pipeline improves the fluency and understandability of model outputs, it further amplifies self-bias. To mitigate such biases, we discover that larger model size and external feedback with accurate assessment can significantly reduce bias in the self-refine pipeline, leading to actual performance improvement in downstream tasks. The code and data are released at https://github.com/xu1998hz/llm_self_bias.

## introduction large language models (llms) have shown strong capabilities in many nlp tasks. while these models still make mistakes, recent studies show that "self-refine" (also known as "self-reflection") is promising to rectify errors based on llm's selffeedback @xcite @xcite @xcite . meanwhile, opposite study also shows that llms fail to correct their mistakes and their performance even gets worse after self-feedback @xcite . these contradictory results suggest that llm's self-feedback is unreliable. selfrefine procedure relies on llm's evaluation capability of the generated text. we hypothesize that if there is a bias during the self-evaluation process, such bias will be amplified during iterative self-figure 1 : how llm's self-feedback inflates scores compared to human assessment. bias is the mean difference between llm and human scores, while skewness (dskew) measures the asymmetry of their distribution around zero. non-biased estimation will have @xmath0. in this work, we define "self-bias" to the degree that an llm favors its own generation. we propose to use two principled statistics to estimate self-bias in llm's self-refinement procedure. the first one measures the degree of inflation in the llm's self-evaluation compared to the true (human) evaluation. the second measures whether llm's self-evaluation is skewed compared to the ture estimate. figure 1 illustrates these two statis-tics. we examine self-bias scores on six diverse llms, covering four languages across three distinct tasks: machine translation, constrained text generation, and mathematical reasoning. we find that self-bias is universal in self-refine and selfrewarding pipelines, regardless of the languages and tasks. this bias causes llms to optimize for false positive corrections rather than improving the actual output quality. we further investigate what is the real benefit of self-refine. we find that while the self-refine pipeline improves the fluency and understandability of model outputs, it does not necessarily lead to intended improvements as specified in the prompt. moreover, llms may favor texts that mirror their style, potentially leading to false positive optimization and reduced diversity in text generation. to mitigate the self-bias, we propose two solutions: increasing the model size and incorporating external feedback to provide accurate assessment, thereby directing the llm towards more accurate self-correction. our contributions are: despite some demonstrations of performance improvements, most findings indicate that llms struggle to rectify their initial mistakes, and their performance even worsens after self-correction @xcite @xcite . this issue arises because the quality of the model's self-generated feedback is bounded by its existing knowledge and abilities @xcite . therefore, internal feedback may not offer any extra advantage for improving the results; it might even steer the model away from the correct answer @xcite . however, prior works only had empirical observations on this phenomenon, while lacking a quantitative analysis. moreover, prior works only focus on specific tasks, such as reasoning or code generation. in this work, we are the first to quantitatively analyze the self-bias of different llms across three tasks and four languages, which provides a novel and generalizable view to address the perils of self-refine. llms as evaluators. liu et al. (2023a) leverages gpt-4 to evaluate text through chain-ofthoughts prompting. fu et al. (2023) leverages gpt-3's sequence likelihood to estimate model performance. kocmi and federmann (2023); xu et al. (2023) designed detailed error schemes for llm to output fine-grained error annotations. despite the popularity of using llms as evaluators, koo et al. (2023) pointed out that llm exhibits cognitive bias when evaluating the text, misaligning from human preference. zheng et al. (2023) pointed out llms have verbosity and self-enhancement bias, which makes them prefer long and verbose answers and answers generated by themselves. chang et al. (2023) found out that llm prefers memorized text over non-memorized text, creating unfair judgments over texts. deutsch et al. (2022); liu et al. (2023b) point out that reference-free metrics are inherently biased on their own outputs. although the above empirical studies provide valuable insights, they lack a formal definition to quantify those biases nor provide a connection to the self-refine framework. in this work, we define and quantify self-bias and provide the first in-depth analysis of its impact on the self-refine pipeline. we analyze potential bias attributions and pinpoint two mitigation directions. ## quantifying self-bias this section outlines the approach used to quantify the self-bias exhibited by llms in an iterative selfrefinement pipeline. we employ statistical bias and distance skewness @xcite estimation to measure self-bias. ## conclusion in this study, we define and quantify self-bias in llms with two principled estimated statistics. our experiments across six llm families, four languages, and three tasks reveal that self-bias is prevalent in self-refine or self-rewarding pipelines. this biased self-feedback leads to false positive objectives, hindering performance improvements during iterative refinement. further analysis reveals that while llm improves fluency and understanding of its generated text, they do not necessarily progress in the intended direction, such as improving quality in machine translation or expanding coverage in concept-to-word generation. instead, llms tend to favor texts that adhere to their inherent styles. finally, our research suggests that larger models are more resistant to self-bias, and incorporating external feedback significantly reduces bias, leading to performance improvements in llms.