Assistant-Guided Mitigation of Teacher Preference Bias in LLM -as-a-Judge (2025)

Abstract: LLM-as-a-Judge employs large language models (LLMs), such as GPT-4, to evaluate the quality of LLM-generated responses, gaining popularity for its cost-effectiveness and strong alignment with human evaluations. However, training proxy judge models using evaluation data generated by powerful teacher models introduces a critical yet previously overlooked issue: teacher preference bias, where the proxy judge model learns a biased preference for responses from the teacher model. To tackle this problem, we propose a novel setting that incorporates an additional assistant model, which is not biased toward the teacher model’s responses, to complement the training data. Building on this setup, we introduce AGDe-Judge, a three-stage framework designed to debias from both the labels and feedbacks in the training data. Extensive experiments demonstrate that AGDe-Judge effectively reduces teacher preference bias while maintaining strong performance across six evaluation benchmarks. .

## introduction llm-as-a-judge refers to the use of large language models (llms), such as @xcite , to evaluate text quality by generating feedback and making evaluative judgments @xcite . unlike traditional manual evaluation or automatic metrics (e.g., bleu), this method offers a cost-effective and scalable alternative, achieving strong alignment with human evaluations when using advanced models like gpt-4 @xcite . consequently, llm-as-a-judge has seen growing adoption in llm evaluation tasks @xcite @xcite . concerns regarding the high cost, limited transparency, and lack of controllability of proprietary large models (e.g., gpt-4) have driven the adoption of proxy judge models based on open-source llms @xcite @xcite . to equip smaller open-source models with evaluation capabilities comparable to those of advanced large models, this approach typically employs a high-performing teacher model-such as gpt-4-to generate judge-specific training data for evaluation tasks. the open-source model is then fine-tuned on this data to create a specialized proxy judge model. by constructing high-quality judge datasets and applying techniques such as weight merging @xcite , the resulting proxy judge models can achieve performance close to that of large models like gpt-4 @xcite , enabling broad practical adoption. despite the advantages of proxy judge models, we identify a key limitation: these models exhibit a significant bias favoring responses generated by the teacher model, regardless of their actual quality. we term this phenomenon teacher preference bias (cf. figure 1 ). this bias stems from the selfpreference bias of the teacher model itself @xcite , which is contained in the training data and subsequently captured by the proxy judge model through fine-tuning. to the best of our knowledge, teacher preference bias has not been previously identified or systematically investigated. our experiments confirm the presence of this bias, which arises from both the judge labels and feedback included in the training data. as gpt-4 is a dominant and highly capable model, many existing methods rely exclusively on data generated by gpt-4 @xcite @xcite , further amplifying the spread of this bias and posing a substantial challenge to the fairness and accuracy of proxy judge models. to mitigate teacher preference bias, we propose to introduce an additional, smaller llm, termed as assistant model, into the training pipeline of proxy judge models. these models do not have a biased preference toward the teacher model's responses, and are relatively inexpensive with decent evaluation capability. within this framework, we leverage the assistant model to debias the teacher modelgenerated data from two sources: biased labels and biased feedback. firstly, we aim to identify and filter out the instances with biased labels leveraging information from the reliable labels agreed by the assistant model. then, we aim to debias the feedback by reducing the teacher model's tendency to overemphasize minor or superficial issues. to this end, we propose a three-stage framework, agde-judge (assistant-guided debiasing for judge models), to mitigate teacher preference bias in proxy judge models. first, we filter out biased labels using an implicit reward margin @xcite , which is derived from consensus labels between the teacher and assistant models. second, we leverage the assistant model to identify severe flaws in responses, thereby counteracting the overemphasis on minor or superficial issues in the feedback. finally, we fine-tune the student model using the refined data to obtain the final proxy judge model. experimental results show that agde-judge effectively reduces teacher preference bias while maintaining high evaluation accuracy across six standard benchmarks. in summary, the main contributions of our paper are as follows: llm-as-a-judge zheng et al. (2023) proposed llm-as-a-judge, leverages powerful llms (e.g., gpt-4) to evaluate responses to open-ended questions, offering a scalable alternative to costly human annotators and limited traditional methods. studies show llms achieve high agreement with human experts @xcite , driving its adoption across various tasks @xcite @xcite . however, this paradigm suffers from biases, including position bias @xcite , verbosity bias @xcite , and self-preference bias, where llms favor their own responses @xcite . these biases undermine the reliability and fairness of llm-as-a-judge evaluations @xcite . proxy judge models due to the high cost, limited transparency, and lack of controllability associated with proprietary large models, finetuning open-source models to serve as judge models-commonly referred to as proxy judge models-has become an increasingly popular alternative @xcite @xcite . this approach typically involves leveraging a powerful proprietary model (mostly gpt-4), known as the teacher model, to construct evaluation-specific training data. however, the self-preference bias implicitly embedded in the teacher model's outputs may be inherited by the proxy judge model, introducing subtler forms of bias. despite its significance, this issue has not yet been systematically studied. debiasing approaches various debiasing approaches have been proposed to address the widespread biases in llm-as-a-judge. discussionbased methods mitigate the bias of a single llm by leveraging multiple llms to engage in debate and deliberation @xcite . adversarial methods introduce structured rationale pairs @xcite , or construct specially curated debiasing datasets @xcite . however, addressing the previously unstudied teacher preference bias calls for new debiasing methods specifically tailored to this issue. ## preliminary study problem definition llm-as-a-judge is an efficient and effective strategy for evaluating llmgenerated responses to compare llm performance. we focus on the most common and effective evaluation setting of pairwise ranking @xcite , where the judge llm evaluates and rank pairs of llm responses. let m denote the judge llm. given an input tuple consisting of a instruction q, two responses r 0 and r 1 generated by llms to be evaluated, and an auxiliary input e which includes evaluation criteria and reference answers @xcite , the judge llm generates a textual feedback v to illustrate the reason behind its judgment, and a preference label @xmath0, 1} to indicate which response is better. formally, due to the limitations of proprietary large judge models (e.g., gpt-4), such as high cost, limited transparency and controllability, researchers have increasingly turned to proxy judge models as a practical alternative. a common approach begins by constructing a training dataset using outputs from the proprietary model, referred to as the teacher model m t , denoted as d @xmath1, r 0 , r 1 , e), (v t , y t )}, where v t and y t are the textual feedback and label generated by m t . an opensource model is then fine-tuned on d t to obtain a proxy judge model, referred to as the student model m s , which is used for evaluation tasks. this method of constructing proxy judge models has demonstrated strong performance comparable to that of the teacher model @xcite . formally, denoting the feedback and label generated from proxy judge model as v s and y s , teacher preference bias despite the advantages of proxy judge models, we identify a critical issue in pairwise ranking: when one response is generated by the teacher model and the other by an unrelated model, the proxy judge often exhibits a biased preference for the teacher model's response, which is much larger than the actual probability of the teacher model's response being better . we refer to this phenomenon as teacher preference bias. this bias stems from the self-preference bias @xcite captured in d t generated by the teacher model who often favors its own responses. denoting the ground-truth label as y @xmath2, 1}, and the response index generated by m t as y @xmath3, 1}, this bias can exhibit as p (y @xmath4. this issue poses a significant challenge to building reliable proxy judge models. first, existing evaluation datasets (e.g., preference collection @xcite , autoj-pairwise @xcite , judgelm @xcite ) heavily rely on the state-of-the-art teacher model gpt-4, which is widely used due to its strong alignment with human preferences @xcite . moreover, the response pairs in these datasets often include outputs generated by gpt-4, which creates the condition for introducing such bias. also, many evaluation benchmarks involve direct comparisons between a gpt-4 response and a response from another model (e.g., mt-bench @xcite , chatbot-arena @xcite , reward-bench @xcite ). as a result, the proxy judge model can easily learn and exhibit this latent bias. in the following section, we systematically demonstrate the existence of this bias. evaluation setup to demonstrate the existence of teacher preference bias, we fine-tune proxy judge models using gpt-4 as the teacher model and evaluate their biased preference toward gpt-4-generated responses. the bias evaluation is primarily conducted on the following datasets: for fine-tuning datasets, we employ autojpairwise and preference collection. both are constructed under gpt-4 with evaluated responses from gpt-4. in addition to using gpt-4 as the teacher model, we investigate the severity of teacher preference bias by introducing an alternative teacher model: gpt-3.5-turbo. we prompt gpt-3.5-turbo with the same inputs used in the fine-tuning datasets to generate feedback and labels. comparing the performance of proxy judge models fine-tuned with gpt-3.5-turbo versus gpt-4 as teacher enables us to evaluate the extent of bias toward gpt-4-generated responses. for base proxy judge model, we use the open-source mistral-7b-instruct-v0.3 @xcite . furthermore, we investigate different sources of bias. we hypothesize that bias may arise from both the preference labels and the textual feedback. to isolate the effect of feedback, we introduce an additional setting where we retain only training instances for which gpt-4 and gpt-3.5-turbo teacher models assign identical preference labels. this allows us to assess the extent of bias attributable specifically to the feedback. more details regarding dataset processing, fine-tuning, and generating procedures are in appendix a. ## methodology after identifying the teacher preference bias, our goal is to mitigate this bias in proxy judge models while maintaining their strong evaluation performance. we incorporate an additional smaller llm, referred to as the assistant model, into the fine-tuning process. the assistant model is costeffective, exhibits reliable evaluation capabilities, and remains unaffected by the teacher model's bias. building on our earlier observations, the proposed framework aims to reduce teacher preference bias by targeting two key sources sequentially: first, by filtering biased labels, and then by addressing biased feedback. we propose agde-judge, a threestage debiasing framework to achieve this goal. ## conclusion this work investigates a previously overlooked bias: teacher preference bias. through extensive experiments and analysis, we demonstrate that training datasets generated by teacher models inher-ently contain self-preference bias in both labels and feedback, which in turn induces teacher preference bias in proxy judge models. to mitigate this, we introduce agde-judge, a three-stage framework that filters biased labels using an implicit reward margin and refines biased feedback with the help of an assistant model. our experimental results show the superior performance of our approach over several existing baselines across multiple standard and bias evaluation benchmarks.