Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM -as-Judge (2025)

Abstract: LLM-as-Judge has emerged as a scalable alternative to human evaluation, enabling large language models (LLMs) to provide reward signals in trainings. While recent work has explored multi-agent extensions such as multi-agent debate and meta-judging to enhance evaluation quality, the question of how intrinsic biases manifest in these settings remains underexplored. In this study, we conduct a systematic analysis of four diverse bias types: position bias, verbosity bias, chain-of-thought bias, and bandwagon bias. We evaluate these biases across two widely adopted multi-agent LLM-as-Judge frameworks: Multi-Agent-Debate and LLM-as-Meta-Judge . Our results show that debate framework amplifies biases sharply after the initial debate, and this increased bias is sustained in subsequent rounds, while meta-judge approaches exhibit greater resistance. We further investigate the incorporation of PINE, a leading single-agent debiasing method, as a bias-free agent within these systems. The results reveal that this bias-free agent effectively reduces biases in debate settings but provides less benefit in meta-judge scenarios. Our work provides a comprehensive study of bias behavior in multi-agent LLM-as-Judge systems and highlights the need for targeted bias mitigation strategies in collaborative evaluation settings.

## introduction large language models (llms) have achieved remarkable performance gains in recent years @xcite @xcite , however, further progress is increasingly constrained by the limited availability of highquality data and the escalating costs of data acquisition and annotation @xcite ; shen top: given responses from two different assistants to the same prompt, the judge model is tasked with selecting the superior response. left: in the multi-agent-debate framework, a judge provides an initial judgment and then debates with a referee. right: in the meta-judge framework, the meta-judge either selects the best judgment or generates its own judgment based on the outputs of multiple judges. @xcite . as task complexity grows, generating the massive amounts of labeled data required through human effort alone has become impractical. to address this bottleneck, recent research has explored using llms themselves as sources of supervision. notably, burns et al. observed that supervision from weaker models has potential to help elicit the capabilities of stronger models. these findings have further motivated the shift toward automated, scalable feedback mechanisms that reduce reliance on costly human annotation. within this line of work, the llm-as-judge paradigm @xcite @xcite @xcite , which uses llms to automatically evaluate responses and provide reward signals, has attracted particular interest due to its adaptability to custom evaluation criteria and its ability to generate human-understandable explanations of decisions. building upon this paradigm, recent efforts have begun to investigate whether incorporating multiagent frameworks-which have proven effective in complex reasoning tasks @xcite @xcite -can further enhance the capabilities of llm-as-judge pipelines. structures such as multiagent debate @xcite @xcite and meta-reasoning @xcite offer promising avenues for boosting evaluation quality. despite these advances, concerns remain. several studies reveal that even single-agent llm judges have been shown to exhibit intrinsic biases, including position bias and verbosity bias, which can undermine the reliability of their evaluations @xcite @xcite . however, there has been limited focused or systematic investigation into whether such biases persist, diminish, or are amplified in multi-agent settings. in response to these concerns, this study systematically examines how intrinsic biases manifest in multi-agent llm-as-judge systems. as demonstrated in figure 1 , we investigate two widely adopted paradigms: llm-as-judge with multi-agent-debate and llm-as-meta-judge. for each paradigm, we also evaluate the performance of different llms as base evaluators. our analysis focuses on four diverse types of bias: position bias, verbosity bias, chain-of-thought (cot) bias, and bandwagon bias, ensuring a comprehensive examination across key dimensions of judgment behavior. interestingly, our results show that intrinsic biases manifest differently across the two multi-agent frameworks. the multi-agent-debate framework amplifies biases sharply after the initial debate, and this increased bias is sustained in subsequent rounds. in contrast, the llm-as-meta-judge approach exhibits greater resistance to these intrinsic biases. when the meta-judge selects the best judgment from a set of candidates, the resulting bias levels are comparable to those observed in single-agent settings. however, when the meta-judge generates a new judgment based on the candidate judgments, a more substantial reduction in bias is achieved. additionally, we examine whether a leading single-agent bias mitigation strategy can be effectively extended to improve evaluation reliability in more complex multi-agent settings, particularly when direct modifications to the judge model and its prompt are impractical or undesirable. to this end, we incorporate pine @xcite , a method shown to eliminate position bias through modifications to position embeddings and causal masking, as a bias-free agent within a multi-agent llm-as-judge system. our results show that introducing a bias-free agent into the debate setting yields consistent improvements as the number of conversation rounds increases. in contrast, its effect in the meta-judge framework remains limited, regardless of whether the metajudge selects an existing judgment or generates its own. these findings underscore the importance of accounting for bias dynamics in multi-agent evaluation and highlight the need for mitigation strategies specifically tailored to collaborative decisionmaking settings. ## conclusion in conclusion, our experiments show that multi-agent debate can amplify intrinsic biases in llms through agent interactions, while the llm-as-meta-judge approach demonstrates better consistency, and sometimes improvement, against these biases, especially with larger judgment pools. notably, the presence of a bias-free agent makes the multi-agent debate framework even more promising, at times surpassing the meta-judge approach. however, these results highlight the importance of developing bias-free agents that can effectively debias within fewer rounds of interaction. overall, our findings offer guidance for deploying multi-agent llm-as-judge frameworks and emphasize the need for universal debiasing techniques capable of addressing multiple biases simultaneously, particularly in collaborative evaluation settings.