O ffset B ias: Leveraging Debiased Data for Tuning Evaluators (2024)

Abstract: Employing Large Language Models (LLMs) to assess the quality of generated responses has become a widely adopted evaluation method. Specifically, instruct-tuned models and fine-tuned judge models based on open-source LLMs have been reported. While it is known that judge models are vulnerable to certain biases, such as favoring longer answers regardless of content, the specifics of these biases remain under-explored. In this work, we qualitatively identify six types of biases inherent in various judge models. We propose EvalBiasBench as a meta-evaluation collection of hand-crafted test cases for each bias type. Additionally, we present de-biasing dataset construction methods and the associated preference dataset OffsetBias. Experimental results demonstrate that fine-tuning on our dataset significantly enhances the robustness of judge models against biases and improves performance across most evaluation scenarios. We release our datasets and the fine-tuned judge model to public.

## introduction language model-based evaluation has become a scalable solution for evaluating text generated by language models. the use of proprietary large language models (llms) such as @xcite as evaluators has demonstrated high correlations with human evaluations @xcite and is increasingly being adopted in llm evaluation benchmarks @xcite . subsequently, concerns regarding cost and reproducibility have led to fine-tuning of open-source models as costeffective judge models @xcite @xcite . although model-based evaluators have shown potential, they often struggle in certain evaluation scenarios, especially with adversarial instances @xcite . judge models are ## bias of judge models we deal with the scenario where a judge model receives an instruction and a pair of good and bad responses and must choose the good response. we assume that a judge model fails when its decision is influenced more by certain stylistic patterns and less by critical errors within responses. figure 1 illustrates such scenario. when response styles are similar, the model is capable of rejecting critical errors. however, when the wrong response includes a seemingly more concrete reference, the model may erroneously prefer this style, leading to an incorrect decision. we refer to these overly preferred patterns as biases. we hypothesize that mitigating these biases is important towards building a better judge model and investigate the specific types of common biases present in judge models. from prior research, it is known that judge models exhibit certain biases such as verbosity bias and self-enhancement bias @xcite . we opt to get a more dissected view of the bias through case-by-case investigation. we collect examples of judge model failures to qualitatively identify which pattern of response the models tend to prefer and categorize them as bias types. in order to discover existing biases, we follow these steps: the internet protocol (ip) is the address on the [...] types of data transfer. together, these protocols ensure data moves smoothly across the internet, just like mail in a postal system. bad response type 1. length bias instruction: is eating orange good for our health? type 2. concreteness bias good response eating oranges is good for well-being. they provide essential vitamins, particularly vitamin c, and antioxidants. according to a study published in the journal of nutrition (2020), eating apples daily can reduce the risk of chronic diseases by 25% bad response instruction: i need some tips on improving my skills. type 3. empty reference bias good response i'd be glad to help! can you specify which skills you're looking to improve? knowing the specific area will allow me to give you tailored advice. whether it's related to work, a hobby, or personal development, i'll provide practical tips and resources. let's make sure we target your goals effectively. to improve your public speaking skills, practice regularly, get feedback from peers, and watch videos of skilled speakers for tips. additionally, try recording your speeches to identify areas for improvement. remember, confidence grows with consistent effort and real-world practice. ## training data construction to reduce biases inherent in judge models, we aim to train them on a collection of counter-examples to biases. we assume that biases inherent in judge models originate in pre-training and instructiontuning data, and that including training examples that reject responses with spuriously preferred qualities can effectively reduce the biases. to achieve this, we build offsetbias, a pairwise preference dataset that aims to complement existing training datasets for judge models. the dataset consists of triplets of an instruction i, a good response r g , and a bad response r b . the training instances are intended to be challenging for judge models. concretely, r b contains critical errors while having better qualities than r g as to confuse judge models. ## discussion coverage of bias types by offsetbias while evalbiasbench focused on bias types, the offsetbias construction methods are not directed to cover all bias types. due to difficulties in data construction described in section 4.1, we had to focus on errors and leave gpt-4 to naturally generate qualities that would manifest some of the biases we identified. the decreased performance of reward model in familiar knowledge bias type in ## conclusion in this work, we identified the biases present in most judge models and categorized them into six 1050 types. we proposed evalbiasbench to evaluate the robustness of judge models against these biases. in addition, we present de-biasing dataset construction methods and the preference dataset offsetbias. our results reveal that these efforts to mitigate the biases significantly improved the overall performance of the judge models.