Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024)

Abstract: Large Language Models (LLMs) have recently been shown to be effective as automatic evaluators with simple prompting and in-context learning. In this work, we assemble 16 LLMs encompassing four different size ranges and evaluate their output responses by preference ranking from the other LLMs as evaluators, such as System Star is better than System Square. We then evaluate the quality of ranking outputs introducing the Cognitive Bias Benchmark for LLMs as Evaluators (CoBBLer), a benchmark to measure six different cognitive biases in LLM evaluation outputs, such as the Egocentric bias where a model prefers to rank its own outputs highly in evaluation. We find that LLMs are biased text quality evaluators, exhibiting strong indications on our bias benchmark (40% of comparisons made by all models) within each of their evaluations that question their robustness as evaluators. Furthermore, we examine the correlation between human and machine preferences and calculate the average Rank-Biased Overlap (RBO) score to be 44%, indicating that machine preferences are misaligned with humans. According to our findings, LLMs may still be unable to be utilized for automatic annotation aligned with human preferences.

## introduction large language models (llms) @xcite adapted to follow various kinds of instructions have been popularly utilized for several natural language tasks. the general standard for testing a model's capabilities is benchmarking its performance on static evaluation suites such as @xcite and @xcite . with the increased usage of language models as general-purpose assistants and their artificial nature @xcite , current task-specific benchmarks are insufficient to measure the quality of generated texts in the wild. recent studies have shown that llms can serve as evaluators themselves: wu and aji (2023) utilize llms as self-evaluators to automatically judge the quality of open-ended generations and compare them with human judgments via an elo-score calculation. other works, such as alpacaeval @xcite , also utilize llms, such as @xcite , as automatic evaluators to reduce the time and cost overhead of human annotations. as noted by these works, such automatic evaluation leaderboards have a number of limitations, including a preference for long outputs or outputs that are more similar to the evaluators' generation qualities. in this work, we propose cobbler, the cognitive bias benchmark for evaluating the quality and reliability of llms as evaluators, as depicted in figure 1 . we collect a set of 50 questionanswering instructions from two well-established benchmarking datasets: bigbench @xcite and eli5 @xcite . we then generate responses from 16 open-and closedsource llms and conduct a round-robin over every possible unique pair between each of the model responses, prompting each model to evaluate its own and other models' responses. we then test six different biases to benchmark their evaluation quality and categorize the model biases into two groups: (1) implicit biases, which can be implicitly extracted from each model's evaluation via a vanilla prompt, and (2) induced biases, which add modifications to the original prompts akin to induce negative behaviors. as shown in figure 2 , we find that the majority of the models strongly exhibit several of the different biases, which may compromise the credibility of their role as evaluators. 2 furthermore, we conduct experiments for human preferences by crowdsourcing six human annotators and collecting each of their rankings for a total of 300 annotations. from our findings, we observe a low correlation between human and machine judgments via rank-biased overlap (rbo), indicating that machine and human preferences are generally in low agreement. our core contributions are as follows: from our benchmark, we find that most models exhibit various cognitive biases when used as automatic evaluators, which may negatively impact evaluation quality. thus, we propose our benchmark (cobbler) for measuring the capabilities of language models as evaluators to enable more reliable evaluations that are well-aligned with human judgment. we note that our use of biased and unbiased preferences does not allude to the ability to make completely impartial judgments but rather the amplification of human-like biases within language models. as most models are tuned on human data, our study aims to estimate this gap between model and human judgment such that they can be refined more effectively to mitigate against these biases. as such, we also aim for our benchmark to be applied towards the development of future models, as in discovering new gaps or finding that existing gaps are still unresolved. ## cobbler: cognitive bias benchmark for llms as evaluators the following criteria are used to select each type of evaluation bias: we carefully hand-select these biases based on the above three criteria such that they can be widely applicable to most evaluation settings in assessing the performance of llms as automatic evaluators. ## experiment setup in this section, we discuss our evaluation framework for benchmarking each of the different biases in llms as evaluators for text quality comparison. ## results and discussion for each bias, we analyze the performance of each of the 16 models as evaluators. we provide a visual breakdown of the proportional impact of the average performance of each model as unbiased evaluators in fig. 2 based on the results relative to the random baseline in ## conclusion in this paper, we analyze 16 recently developed llms for their suitability as automatic text quality annotators in q/a settings. we introduce a new benchmark cobbler to assess their evaluation performance against 1) implicit and 2) induced biases. additionally, we compare llm evaluations to human preferences and find only a 44% average agreement. our results indicate that most llms exhibit cognitive biases to a greater extent than humans, suggesting that llms are still unsuitable as fair and reliable automatic evaluators. in the future, potential de-biasing methods provide another area of interest in reducing each bias. for example, techniques such as chain-of-thought (cot) reasoning or other alignment methods can perhaps be employed to reduce the bias for current models.