Mitigating the Bias of Large Language Model Evaluation (2024)

Abstract: Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality. However, existing judges are proven to be biased, namely they would favor answers which present better superficial quality (such as verbosity, fluency) while ignoring the instruction following ability. In this work, we propose systematic research about the bias of LLM-as-a-Judge. Specifically, for closed-source judge models, we apply calibration to mitigate the significance of superficial quality, both on probability level and prompt level. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality. We apply our methods on the bias evaluation benchmark, and experiment results show our methods mitigate the bias by a large margin while maintaining a satisfactory evaluation accuracy.

## introduction recently, the evaluation for large-scale language models (llms) has drawn considerable attention of research community @xcite . as the capabilities of llms continue to develop across various tasks, it is essential to evaluate them from a comprehensive perspective @xcite . however, traditional evaluation metrics for generative models, such as bleu @xcite and rouge @xcite , only capture limited aspects of a model's performance. some research has proposed llm-as-a-judge @xcite , namely utilizing proprietary llms, especially gpt-4 @xcite , to evaluate the llm's response. however, relying on external api for evaluation may introduce consideration about privacy leakage. other works propose to fine-tune open-source models as external evaluators @xcite . they construct evaluation data and fine-tune llms to endow them specialized evaluation ability comparable with gpt-4. however, the fine-tuned judges underperform gpt-4 in terms of fairness and scalability by a large margin @xcite . as a pioneering research, llmbar @xcite introduces the first meta-evaluation benchmark of llm evaluation bias. the benchmark is adversarially designed to test whether llm evaluators can detect instruction-following outputs. they apply their benchmark on existing llm evaluators, and reveal that even state-of-the-art llm evaluators struggle to provide unbiased evaluation on their benchmark. however, their study is mainly based on closed-source models, and they only provide several suggestions to mitigate the bias by prompt engineering. in this work, we propose a systematic framework to mitigate the bias of llm evaluators. for closedsource llm judges, we propose to mitigate the bias by calibration. as the evaluation bias is generally presented as a higher score for superficial quality, we propose two ways to directly model the superficial quality, and subtract it from the final result. more specifically, for probability-based evaluators, we propose to model the superficial quality by utilizing the probability generated by the pre-trained model. for generation-based evaluators, we propose to model the superficial quality by probing the superficial attributes in the prompt template. for open-source llm evaluators, we propose to mitigate the bias of llm evaluators by pairwise contrastive training. delicately curated negative samples are included in the training set where the answer deviates from the instruction but presents better superficial quality. we evaluate our methods on the bias benchmark of llmbar, which consists of one natural and four adversarial test sets to benchmark llm evaluators' ability to detect instruction-following answers unbiasedly. experiment results show that all of our proposed methods mitigate the evaluation bias on the adversarial sets by a large margin, while keeping a comparable evaluation performance on the natural set. ## related work presently, evaluation research on llms can be grouped into three distinct methodologies: human evaluation, benchmark evaluation, and llm-as-a-judge. the most straightforward approach to evaluating an llm involves engaging dedicated evaluators or crowdsourcing to compare and assess the model's output @xcite @xcite . however, human evaluation is labor-intensive and challenging to scale during the model development cycle. additionally, human subjective factors can influence the evaluation results, making it difficult to guarantee the consistency and stability of model evaluations. other scholars have devised standard test sets for the automatic evaluation of llms. these test sets typically encompass a range of tasks, assessing the llm's performance by comparing its task outputs to standard answers @xcite @xcite . however, the benchmarks typically consist of problems of selection, which are not adapted to quantify the generation ability of llms. given the various issues with benchmark evaluation, some scholars have introduced the evaluation scheme with the name of llm-as-a-judge, which involves utilizing another proprietary llms to assess the output of the current model. for example, alpacaeval @xcite constructs an evaluation set comprising 805 question-answer pairs and leverages gpt-4 to compare the winning rate of the model's answers against text-davinci-003. similarly, mt-bench @xcite creates 80 high-quality multi-round test questions spanning eight common areas, including mathematics and reasoning. it then automatically grades the model's answers using gpt-4. however, relying on an external model to perform evaluation may introduce concerns about the privacy leakage. also, the iterative versions of the api model also make the evaluation inconsistent. to tackle these challenges, some studies have advocated for the training of localized agent evaluation models. for instance, pandalm @xcite constructs data based on alpaca instructions and gpt-3.5 annotation, and then fine-tunes llama-7b @xcite as a judge model. judgelm @xcite constructs data from gpt-4 annotations and fine-tunes a scalable judge model. despite the llm-as-a-judge becoming the default paradigm for llm evaluation, various work has pointed out that the llm-based judges are severely biased, namely the evaluator would make prediction based on spurious features (mainly the superficial quality) while ignoring the alignment with the instruction. various biases are discussed in previous work, including position bias @xcite , verbosity bias @xcite , formality bias and self-enhancement bias @xcite , etc. as a pioneering work, llmbar proposes the first systematic evaluation of llm evaluation bias. they created four adversarial test sets as the testbed for llm's ability of making unbiased prediction. their results indicate that llm evaluators, even gpt-4, suffer severely from the over-reliance on the superficial quality. however, they only propose to compensate for the bias by prompt engineering, which can hardly be applied to fine-tuned judges. a more thorough bias mitigation framework is of urgent demand. ## conclusion in this work, we propose two methods for mitigating the bias of llm-as-a-judge. for closed-source judge models, we propose to mitigate the bias by calibration. for open-source judge models, we propose to mitigate the bias by pairwise contrastive training. we apply our methods on the llmbar benchmark, effectively mitigating the bias of llm evaluation while maintaining a comparable performance. as the llm-as-a-judge becomes a paradigm for evaluating the llm performance, it is advisable to exert more alert on the bias of llm-based evaluators. in the future, we will dedicate more effort to the bias mitigation of llm-as-a-judge into more fine-grained categories. we will also conduct research about the inference bias on other llm-based applications.