LLM s are Biased Evaluators But Not Biased for Fact-Centric Retrieval Augmented Generation (2025)

Abstract: Recent studies have demonstrated that large language models (LLMs) exhibit significant biases in evaluation tasks, particularly in preferentially rating and favoring self-generated content. However, the extent to which this bias manifests in fact-oriented tasks, especially within retrieval-augmented generation (RAG) frameworks—where keyword extraction and factual accuracy take precedence over stylistic elements—remains unclear. Our study addresses this knowledge gap by simulating two critical phases of the RAG framework. In the first phase, LLMs evaluated human-authored and model-generated passages, emulating the pointwise reranking phase. The second phase involves conducting pairwise reading comprehension tests to simulate the generation phase. Contrary to previous findings indicating a self-preference in rating tasks, our results reveal no significant self-preference effect in RAG frameworks. Instead, we observe that factual accuracy significantly influences LLMs’ output, even in the absence of prior knowledge. These findings are consistent among three common QA datasets (NQ, MARCO, TriviaQA Datasets) and 5 widely adopted language models (GPT-3.5, GPT-4o-mini, Gemini, LLaMA3, and Mistral). Our research contributes to the ongoing discourse on LLM biases and their implications for RAG-based system, offering insights that may inform the development of more robust and unbiased LLM systems.

## introduction retrieval-augmented generation (rag) frameworks provide a promising approach to address the challenges of hallucination and outdated training data in classical large language model (llm) prompting @xcite . by integrating information retrieval with generative capabilities, rag frameworks significantly improve the accuracy and relevance of generated content @xcite . nonetheless, recent studies @xcite @xcite have showed that llms tend to favor their own self-generated passages, potentially introducing biases into the outputs. as llm-generated content becomes increasingly prevalent on the web, it is crucial to understand the potential impacts of these biases, especially how they might affect rag-based question-answering (qa) systems in the future @xcite . as the volume of llm-generated material online continues to grow, in rag scenarios, a model may retrieve a mix of human-written and modelgenerated passages. figure 1 illustrates a potential issue: if llms preferentially cite their own content, especially non-factual passages, this bias could degrade the performance of qa systems. this concern gives rise to three research questions: llms preferentially refer to self-written content? how do llms prioritize the factuality, order, and style of a passage? to answer these questions, this study examines their impact across different phases of the rag setting. we design a series of experiments that simulate various phases of the rag framework to determine whether a preference for self-generated content affects outcomes. specifically, we utilize a direct evaluation approach where llms rank the suitability of passages for answering specific questions, mimicking the pointwise reranking phase-a common technique in rag frameworks aimed at enhancing performance @xcite . additionally, we employ a pairwise reading comprehension method to evaluate how llms respond to questions using both selfgenerated and externally sourced passages, analogous to the generation phase of the framework. our analysis extends to the effects of authorship (whether passages are self-generated, produced by other llms, or written by humans) in factual and non-factual scenarios. contrary to previously reported self-preference in passage ranking tasks @xcite , our extensive experiments reveal no overarching self-preference in the final generations of the rag framework, neither in the pointwise reranking nor the pairwise reading phase. this may be due to the model adopting different mindsets when performing different tasks, which will be elaborated in section 5. our contributions are 4-fold: ## related work with the growing popularity of llms @xcite , research on automatic evaluation of llms has also attracted attention. the predominant approach utilizes an llm as the evaluator, which has been shown to be highly correlated to human evaluation, while being significantly cheaper and faster @xcite @xcite . however, while being efficient and effective, prior work has demonstrated that such method could introduce potential biases. @xcite identified several biases and limitations of the llm evaluators, including position bias, verbosity bias, and self-enhancement bias, i.e., preferring responses generated by themselves. @xcite showed that llm evaluators are sensitive to the order and proposed a calibration framework to mitigate the bias. @xcite analyzed llms' multilingual evaluation capabilities and showed that llms underperform on lowresource languages. @xcite showed that llms might favor style over factuality, i.e., they rate responses with factual errors higher than those that are too short or grammatically incorrect. chen et al. (2024) investigated various biases of llm evaluators and demonstrated that this vulnerability could be exploited by malicious attackers. @xcite identified significant verbosity bias on the alpacaeval benchmark @xcite and proposed a length-controlled benchmark as a mitigation. @xcite introduced a cognitive bias benchmark for llm evaluators and found out that they are misaligned to human judgements. @xcite identified that llm evaluators can recognize their own generations and rate them higher. @xcite demonstrated that the popular self-refinement approach could further amplify llms' self-preference. furthermore, @xcite showed that neural retrievers also present biases towards llm-generated texts. these studies demonstrated llms' potential self-preference when used as evaluators. our work extends the exploration of llm's self-preference to the rag framework and provide a thorough result, llm refers to gptgenerated passage chosen from passage bank stage 2. experiments & evaluation stage 1. data construction passages false answers human-written passages (𝑝 𝐻 ) llm rewriting model-generated passages model generated passage (𝑝 𝑚 ) ground-truth (gt) answer fake information false answers • semantically different • grammatically similar to gt phrases, e.g., passage bank (passage defined by (passage, factuality) pair) ground truth (𝐺𝑇): 2005 false answer (𝐹𝐴): 1950 (𝑝 𝑚 , 𝐺𝑇) (𝑝 𝑚 , 𝐹𝐴) (𝑝 𝐻 , 𝐺𝑇) (𝑝 𝐻 , 𝐹𝐴) llm experimental flow experimental flow evaluation evaluation question of interest does the relevance score assigned by llms to selfauthored passages significantly greater than the score of passages authored by others? question of interest do llms more frequently reference self-authored passages when generating responses? chosen from passage bank which shows that llms exhibit behaviors contrary to the previous findings under rag settings. ## methodology this section explains our methodologies and introduces the notations used to evaluate the selfpreference effect of llms in rag settings. we denote a query as q, passages written by humans as p human and a passage written by model m as p m . our proposed experimental framework is illustrated in figure 2 . ## experiments this section presents the results of our two-phase experimental framework design to simulate prevailing rag systems: the pointwise reranking phase and the generation phase. we follow a structured analysis to examine selfpreferential bias in llms across different stages and conditions: ## discussion in this section, we discuss our findings, compare them to prior research, and provide possible explanations for the observed behaviors. our results are consistent with previous work showing that when llms are asked to evaluate passages based on writing quality-a subjective task-they exhibit a clear self-preference bias. however, this bias is much less evident in the pointwise-reranking phase of the rag framework, where the task shifts to assessing the objective relevance of a passage to a given query. in these cases, the primary evaluation criterion becomes factual content rather than stylistic features. we hypothesize that this task-dependent behavior arises from a change in "mindset," or more specifically, from the model leveraging different internal mechanisms when processing subjective versus objective tasks. for instance, stylistic evaluations may engage different decoding paths or attention patterns than factual relevance assessments. future work could investigate this further through mechanistic interpretability tools, such as attention attribution, decoding traces, or internal token-level likelihoods-to better understand how these taskdependent behaviors arise. interestingly, figure 2 shows that self-generated passages receive lower scores than human-written ones in the relevance evaluation, suggesting a weak or even negative self-preference bias. one possible explanation is that supervised fine-tuning (sft) and alignment processes explicitly train models to prefer high-quality human responses, potentially leading them to discount their own generations. additionally, calibration mechanisms such as logit scaling or temperature adjustments may further suppress model confidence, making them less likely to favor their own output, especially when presented alongside human-written responses. in the generation phase, we observe a consistent hierarchy of bias: factuality @xmath0. this ranking is also evident in the pointwise-reranking phase. similar to how students prioritize factual correctness and keyword extraction in reading comprehension tasks, llms exhibit the strongest preference for factual passages. order bias follows, as models often rely on the first plausible answer encountered, consistent with prior work. self-preference is the weakest of the three, especially in tasks where stylistic quality is irrelevant to success. importantly, these findings are robust across all three qa datasets, five llms, and evaluations involving human-written content. this consistency strengthens the generalizability of our observations. overall, our results suggest that hallucinated content generated by llms is unlikely to strongly influence future generations within rag workflows-especially in factual tasks where relevance, not writing style, is the main evaluation criterion. detailed prompts and additional analysis can be found in appendix b.1. ## conclusion our study presents the first comprehensive investigation into the potential impact of self-preference on the performance of rag systems. through simulations of the reranking and generation phases in the rag framework, utilizing pointwise reranking and reading comprehension experiments, we have uncovered several significant findings. first, in general, llms exhibit minimal selfpreference when referring to external resources within rag frameworks. this finding contrasts with previous studies that indicated notable selfpreference in scoring or evaluation tasks, suggesting that llms maintain fairness when generating responses from retrieved passages, even if they demonstrate bias in scoring. second, our experiments reveal a pronounced preference among llms for factual information, being able to reference factual passages rather than stylistically preferential self-generated content, even in cases where they do not possess prior knowledge. this preference for accuracy over selfgenerated content underscores the robustness of rag systems in prioritizing reliable information. our findings mitigates concerns about potential biases affecting system performance and inspires future work on model-specific behavior and taskspecific behavior, and how factors like architectures, datasets, and knowledge levels influence model biases. third, our findings are robust among 5 widely used models and 3 real-world qa datasets. this mitigates concerns about potential biases affecting system performance. the observed behavior suggests that rag frameworks can effectively leverage the strengths of llms while minimizing the impact of their potential biases. finally, our work points to the value of mechanistic interpretability in understanding such behavioral differences. future research analyzing attention distributions, decoding dynamics, or tokenlevel confidence (e.g., log-likelihoods) may reveal how and why llms shift behavior across different task formats. in addition, exploring whether prompt engineering-such as more sophisticated or domain-specific prompt designs-can modulate self-preference presents an important direction. this invites deeper inquiry into how architectures, training data, alignment objectives, and prompt design interact to shape model-specific and taskspecific biases.