Humans or LLM s as the Judge? A Study on Judgement Bias (2024)

Abstract: Adopting human and large language models (LLM) as judges (*a.k.a* human- and LLM-as-a-judge) for evaluating the performance of LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results. In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bias**, **Authority Bias** and **Beauty Bias** on LLM and human judges. We curate a dataset referring to the revised Bloom’s Taxonomy and conduct thousands of evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases. We further exploit these biases to conduct attacks on LLM judges. We hope that our work can notify the community of the bias and vulnerability of human- and LLM-as-a-judge, as well as the urgency of developing robust evaluation systems.

## introduction proprietary models such as @xcite , claude (anthropic, 2024), gemini-pro @xcite , showcase their outstanding ability in numerous nlp tasks, meanwhile serving as daily-used tools in diverse scenarios. in the meantime, the open-source community is trying to replicate the proprietary models and democratize llms. to better keep track of llm advancement, the community attaches great importance to evaluating model performance by developing numerous * first two authors contributed to this work equally. benyou is the corresponding author. 1 our data are released at https://github.com/ freedomintelligence/humans_llms_judgement_bias benchmarks, which can be roughly categorized into open-ended and close-ended ones. although closeended benchmarks such as mmlu @xcite , c-eval @xcite are convenient to evaluate on, they often suffer from data contamination issue. proprietary llms, which are trained with in-house data, tend to perform particularly well in close-ended benchmarks. on the other hand, open-ended benchmarks (e.g., mt-bench @xcite and @xcite ) test models via free-form generation, which is more consistent with real-world use cases and relies heavily on llms' generation ability. the data contamination issue in open-ended benchmarks is less severe since there are no standard answers, and even with contamination it offers minimal assistance to performance hacking. open-ended benchmarks often count on human to evaluate the answer quality. as the recent emergence of human-aligned llms, llm-as-ajudge @xcite , serves as an alternative to human judges. more recently, both types of judges are found to posses certain biases @xcite , questioning the validity of human-and llm-as-a-judge. therefore, an important question rises: how biased are humans and llms on judging open-ended generation? current bias evaluation frameworks necessitate a golden standard, either in the form of groundtruth (e.g., correct vs erroneous, harmful vs non-harmful) or human providing reference answers. but what if we intend to probe the effect of some perturbations for which the golden standards are not provided or not well defined? in this paper, we first identify the four biases of interest: misinformation oversight bias, gender bias , authority bias and beauty bias, which are crucial in natural language generation (nlg) evaluation. inspired by intervention study, we investigate these biases by adding 4 perturbations (factual error, gender-biased content, fake references and rich content) to raw answers, respectively. to fill the gap of current research, we propose a novel reference-free framework for bias evaluation on human and llm judges. we first form a control group and an experimental group, where each sample in the former contains a pair of answers to the same question, and each answer pair in the latter consists of an answer from the former, and the perturbed version of the other answer. we then quantify the preference shift between the two groups by attack successful rate (asr), where a higher value indicates a judge possessing more severe biases. we further exploit the uncovered biases to perform attacks on llm judges. in summary, our key contributions and findings are summarized as follow: ## experimental protocol in this section, we elaborate on our motivation, experimental methodology, the creation of experimental data, the experimental procedure, evaluation metrics, and the models under evaluation. ## conclusion in conclusion, we develop a novel reference-free framework to explore misinformation oversight bias, gender bias, authority bias and beauty bias in human and llm judges, providing deeper insights into their innate biases and vulnerabilities. we reveal that all judges display significant biases, but diverge in their specific inclinations. additionally, we show the llms' judgement can be hacked via a prompt-based method that we discover. through our work, we hope to provide insights on the bias of human-and llm-as-a-judge, and to notify the community about the urgency of developing more robust evaluation systems.