Large Language Models are not Fair Evaluators (2024)

Abstract: In this paper, we uncover a positional bias in the evaluation paradigm of adopting large language models (LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. We propose a simple yet effective calibration framework to address our discovered positional bias.To evaluate the effectiveness of our framework, we manually annotate the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark’s question prompt. Extensive experiments demonstrate that our approach successfully alleviates evaluation bias, resulting in closer alignment with human judgments.

## introduction the rapid advancement of large language models (llms) @xcite has underscored the importance of evaluating their alignment with human intent in generated responses, making it an active field of research. traditional n-gram metrics like bleu @xcite and rouge @xcite , as well as more sophisticated model-based evaluations such as bertscore @xcite and bartscore @xcite , are insufficient for thoroughly assessing this alignment @xcite . while human evaluation provides 1 https://github.com/i-eval/faireval ## experiments 3.1 human annotation to assess the effectiveness of our proposed strategies, three of the authors manually annotate the "win/tie/lose" outcomes of responses from chat-gpt and vicuna-13b independently in all 80 vicuna benchmark questions. all of the annotators are researchers familiar with artificial intelligence and are well-equipped to assess the quality of the responses. following the same template as the original vicuna, the annotators are instructed to assess the responses provided by vicuna-13b and chat-gpt from four different perspectives: helpfulness, relevance, accuracy, and level of detail. the responses of vicuna and chatgpt are presented to the annotators in random order. the evaluation process for each example took an average of three minutes. the final result is based on the majority opinion among three annotators. ## related work large language models as evaluators how to achieve reliable evaluations of generative ai assistants is one of the most important problems in ai research. traditional metrics, like bleu, rouge, bertscore, or bartscore fail to measure the alignment of the generated response with human intent @xcite . considering the potent capabilities of llms, researchers have started utilizing llms to evaluate the proficiency of generative models in adhering to human instructions @xcite @xcite . there are two typical llm-as-evaluator variations: pointwise evaluation and pair-wise evaluation. pointwise evaluation scores the responses one by one, while pair-wise evaluation achieves the evaluation results by comparing two responses. point-wise evaluation is more convenient, because para-wise evaluation may lack scalability when the number of responses increases. however, pair-wise evaluation can give more accurate evaluation results than point-wise evaluation @xcite . specifically, assigning a precise score to a response is challenging as it requires providing evaluators with a detailed and accurate scoring standard that is difficult to establish. in contrast, just making the comparison is much easier and more stable. in this work, we discover and address the positional bias of llms-as-evaluators for pair-wise evaluation. we believe it is essential for the whole community to achieve reliable evaluations. bias of deep neural networks deep neural networks have been proven to easily learn biases from the data, which significantly impacts their reliability. specifically, bias has also been investigated in natural language inference @xcite @xcite ), question answering @xcite , roc story cloze @xcite , lexical inference @xcite , visual question answering @xcite , information extraction @xcite @xcite and so on. llms are pre-trained using a vast amount of data from the internet, making it highly likely for them to learn biases present in those materials. although the llms are already widely adopted as a proxy of human evaluators, the reliability of this paradigm is not well explored. in this paper, we critically examine the llms-asevaluator paradigm and uncover a significant positional bias. furthermore, we propose three simple yet effective methods to calibrate the positional bias to achieve reliable and fair evaluation results. ## conclusion in this paper, we reveal a positional bias of the llms-as-evaluators paradigm, i.e., by manipulating the order of candidate responses during evaluation, the quality ranking results can be significantly influenced. the positional bias seriously affects the reliability of the evaluation. we propose a calibration framework with three simple yet effective strategies to address the discovered bias. we also manually annotate the "win/tie/lose" outcomes of responses from chatgpt and vicuna-13b in the vicuna benchmark's question prompt to evaluate the effectiveness of our framework. extensive experiments demonstrate that our approach successfully alleviates evaluation bias, resulting in closer alignment with human judgments.