Curse of Knowledge: Your Guidance and Provided Knowledge are biasing LLM Judges in Complex Evaluation (2025)

Abstract: As large language models (LLMs) grow more capable, they face increasingly diverse and complex tasks, making reliable evaluation challenging. The paradigm of LLMs as judges has emerged as a scalable solution, yet prior work primarily focuses on simple settings. Their reliability in complex tasks—where multi-faceted rubrics, unstructured reference answers, and nuanced criteria are critical—remains understudied. In this paper, we constructed ComplexEval Bench, a challenge benchmark designed to systematically expose and quantify Auxiliary Information Induced Biases. We systematically investigated and validated 6 previously unexplored biases across 12 basic and 3 advanced scenarios. Key findings reveal: (1) all evaluated models exhibit significant susceptibility to these biases, with bias magnitude scaling with task complexity; (2) notably, Large Reasoning Models (LRMs) show paradoxical vulnerability. Our in-depth analysis offers crucial insights for improving the accuracy and verifiability of evaluation signals, paving the way for more general and robust evaluation models.

## introduction as large language models (llms) face increasingly complex and diverse tasks @xcite , the evaluation of these tasks becomes correspondingly more challenging (e.g., mathematical proofs, creative writing). the llm-as-ajudge paradigm offers a low-cost and highly automated solution @xcite , demonstrating strong alignment with human experts @xcite . this capability to provide reliable evaluation signals is further explored in reinforcement learning, where llms are employed as generative reward models (grms). while llm judges have the potential to address the growing demand for reliable and verifiable eval-figure 1 : "when help become harm": the figure shows that auxiliary information improves llm judges but introduces new biases in complex evaluation uation signals in complex tasks @xcite , existing methods often fall short. evaluating complex tasks often requires more guidance and provided knowledge @xcite . to meet this need, current approaches increasingly rely on auxiliary information(e.g., reference answers @xcite , rubrics @xcite @xcite ), which can be external or self-generated. the use of auxiliary information enhances evaluation reliability and verifiability, however, it also introduces new challenges: llm judges often struggles to follow complex instructions @xcite and can be influenced by specific patterns @xcite .previous studies have identified various evaluation biases @xcite ), but they have generally failed to address these complex and information-rich evaluation tasks. we therefore investigate a crucial yet unexplored question: does the use of auxiliary information introduce bias in complex evaluation? in this paper, we study bias in complex evaluation, where reference,rubrics and background knowledge fundamentally shape judgment quality. we propose complexeval, an adversarial framework designed to diagnose auxiliary information induced biases. the framework's two-tiered structure exposes these biases at varying levels of granularity: 1) complexeval-basic conducts freeform attacks, broadly exploring bias induced by auxiliary information across twelve general tasks, while 2) complexeval-advanced conducts fixedpattern attacks, deeply analyzing specific bias types to study how rubrics, references and other auxiliary information corrupt evaluation mechanics based on three complex evaluation tasks. our analysis reveals three key insights about auxiliary information induced bias: 1) all models exhibit significant biases when incorporating rubrics and reference answers, with severity scaling with task complexity; 2) we identify five specific bias patterns, including format bias, solution fixation bias, stereotype amplification bias, criteria loophole bias and criteria entanglement bias; 3) while advanced reasoning models usually achieve better scores, they show greater vulnerability to these auxiliary information induced bias. our key contributions are as follows: ## conclusion in summary, this study formally identifies and validates a category of auxiliary information induced biases previously unexplored in prior research, demonstrating how these biases undermine the reliability of large language model evaluations in complex scenarios. we further conduct finegrained analysis of six specific bias types, revealing their essential nature stems from models' limited capacity to effectively utilize auxiliary information. through this work, we alert the research community to the critical severity of evaluation biases in complex scenarios, and aim to provide foundational insights for advancing llm-as-a-judge capabili-ties toward more sophisticated and general-purpose evaluation settings.