Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments (2024)

Abstract: Large language models (LLMs) have shown promising abilities as cost-effective and reference-free evaluators for assessing language generation quality. In particular, pairwise LLM evaluators, which compare two generated texts and determine the preferred one, have been employed in a wide range of applications. However, LLMs exhibit preference biases and worrying sensitivity to prompt designs. In this work, we first reveal that the predictive preference of LLMs can be highly brittle and skewed, even with semantically equivalent instructions. We find that fairer predictive preferences from LLMs consistently lead to judgments that are better aligned with humans. Motivated by this phenomenon, we propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework, ZEPO, which aims to produce fairer preference decisions and improve the alignment of LLM evaluators with human judgments. To this end, we propose a zero-shot learning objective based on the preference decision fairness. ZEPO demonstrates substantial performance improvements over state-of-the-art LLM evaluators, without requiring labeled data, on representative meta-evaluation benchmarks. Our findings underscore the critical correlation between preference fairness and human alignment, positioning ZEPO as an efficient prompt optimizer for bridging the gap between LLM evaluators and human judgments.

## introduction large language models (llms) @xcite @xcite have become the standard machinery for evaluating the quality of natural language generation over various aspects, such as coherence, fluency, and truthfulness, in a reference-free manner @xcite @xcite . figure 1 : illustration of the zepo pipeline. given a manual prompt, the distribution of llm preferences can be biased towards a certain class. zepo optimizes the prompt on a zero-shot fairness learning objective until the balance is achieved in the distribution. owing to the remarkable in-context learning capabilities of llms @xcite , prompting techniques further enable versatile use of llm evaluators with user-defined evaluation criteria, where pairwise-preference-based evaluators have so far demonstrated superior human alignment to direct scoring @xcite . however, llms have been known to exhibit preference bias @xcite , a priori propensity to predict certain classes over others unfairly, and display strong sensitivity to the actual prompts describing evaluation criteria @xcite . the preference bias is argued to be largely due to various factors that result in a label distribution shift, such as position bias @xcite , verbosity bias @xcite , and contextual bias @xcite , where llms unfairly favor later and longer answers, or even follow repetitive answers in their demonstrations. we are thus motivated to explore the impact of preference biases on human alignment in the novel context of llm evaluators. we start by conducting a systematic study examining the sensitivity of llm evaluators to the provided instructions. by paraphrasing from a set of instructions, we find that the pairwise preference of llms largely varies even with semantically equivalent instructions, and different instructions exert different degrees of preference biases. noticeably, we observe that fairer preferences consistently lead to better human-aligned judgments. motivated by this empirical finding, we then propose an automatic zero-shot evaluationoriented prompt optimization (zepo) framework for steering llm evaluators towards better agreements with humans; see fig. 1 . we design a new zero-shot fairness objective function by measuring the absolute difference between a uniform prior distribution and the model preference distribution. zepo, without any labeled data, shows substantial performance gains over state-of-the-art llm evaluators with manually designed instructions on meta-evaluation benchmarks. in sum, we provide the following contributions. 1) we present a systematic analysis that reveals the strong sensitivity of llm evaluators to instructions. importantly, we find that fairer preferences elicit better human-aligned llm judgments. 2) we introduce a zero-shot evaluation-oriented prompt optimization framework (zepo) for automatically optimizing llm evaluators toward better human agreements without any labeled data. 3) we demonstrate that zepo efficiently discovers the fairest instruction for llm evaluators, delivering substantial gains in evaluation over representative tasks. ## fairer preferences elicit improved human-aligned judgments prompt sensitivity and bias. we start by analyzing the sensitivity of llm evaluators to variations in instructions. formally, given some source text and corresponding response candidates as an input query x i , we have the predicted label y i as the model preference. evaluation instruction i is formulated with the input query x i in a prompt template to form a complete context c(x i , i) = template(x i , i) for evaluation. llm evaluators then make predictions by y @xmath0, where the verbalizer y defines the set of preferences (i.e., a or b for pairwise preferences). to inspect prompt sensitivity, we leverage gpt-3.5 (openai, 2023) to generate a set of semantically equivalent instructions @xmath1, ..., i m } by paraphrasing from an initial instruction i 1 . in fig. 2 , we observe a severe fluctuation in human agreement scores by prompting llama-3 8b @xcite model with c @xmath2, i m ). this reflects a high prompt sensitivity and poor robustness of standard llm evaluators. the observation aligns with previous research in position biases @xcite , and llms are sensitive to orders and formats of provided exemplars @xcite . ## zepo: zero-shot prompt optimization with fairer preferences zero-shot fairness learning. motivated by these findings, we now propose to automatically optimize the evaluation prompts for llm evaluators toward fairer preferences, thereby achieving better human alignments. importantly, the source preference distribution for an unbiased pairwise evaluator should naturally be uniform p @xmath3. consequently, algorithm 1 zepo. in an unsupervised set of data d by measuring the absolute difference between the source prior and preference distribution. ## conclusion we first analyzed the relationship between preference fairness and human alignment; it revealed that llm evaluators produce highly skewed preference distributions even with semantically equivalent instructions. we further showed that fairer preferences can yield improved human-aligned llm judgments. based on this insight, we proposed a zero-shot prompt optimization framework with a fairness-aware zero-shot proxy. it substantially improves alignments of pairwise llm evaluators with humans, without any labeled data, and serves as a meta-method orthogonal to debiasing approaches.