D ecipher P ref: Analyzing Influential Factors in Human Preference Judgments via GPT -4 (2023)

Abstract: Human preference judgments are pivotal in guiding large language models (LLMs) to produce outputs that align with human values. Human evaluations are also used in summarization tasks to compare outputs from various systems, complementing existing automatic metrics. Despite their significance, however, there has been limited research probing these pairwise or k -wise comparisons. The collective impact and relative importance of factors such as output length, informativeness, fluency, and factual consistency are still not well understood. It is also unclear if there are other hidden factors influencing human judgments. In this paper, we conduct an in-depth examination of a collection of pairwise human judgments released by OpenAI. Utilizing the Bradley-Terry-Luce (BTL) model, we reveal the inherent preferences embedded in these human judgments. We find that the most favored factors vary across tasks and genres, whereas the least favored factors tend to be consistent, e.g., outputs are too brief, contain excessive off-focus content or hallucinated facts. Our findings have implications on the construction of balanced datasets in human preference evaluations, which is a crucial step in shaping the behaviors of future LLMs.

## introduction human judgments are widely used in summarization evaluation, complementing automatic metrics. automatic metrics often rely on reference texts. they compare system summaries with reference texts based on word overlap, as seen in bleu and rouge @xcite , or contextualized embeddings, as in bertscore, mover-score, bleurt, unieval @xcite @xcite . by contrast, human evaluation usually does not require reference texts. evaluators are asked to assess the quality of summaries along certain linguistic and content dimensions; they can also rank outputs from various systems @xcite @xcite . this type of reference-free evaluation has proven beneficial, especially as systems are increasingly matching the performance of humans. pairwise human judgments have become extensively used in recent years for the development of large language models @xcite @xcite @xcite @xcite ). instructgpt, for example, learns a reward model from human comparisons, then optimizes against this reward model to generate outputs that align with human preferences. 1 human ratings are further used in the final evaluations to rank model variants based on specific prompts and their completions @xcite . models with higher winrates are considered to have an edge. human comparisons have been chosen for their ease and intuitiveness, and they play a critical role in shaping the behavior of llms. despite their significance, there has been limited research probing human preference judgments. using summarization as a case study, we examine the characteristics of system outputs that may influence human judgments. we have selected summarization as our case study, because summaries need to be anchored in their original texts, which facilitates our evaluation of content accuracy. it is worth noting that our proposed framework could potentially be extended to other scenarios. evaluators of summarization and text generation systems are typically asked to consider a range of factors: informativeness, factuality, fluency, coherence, extractiveness, non-redundancy, etc. @xcite @xcite . however, the aggregate effect and relative importance of these factors remain elusive. e.g., factual errors can greatly undermine a system's output. an inaccurate claim made by google's bard about the james webb space telescope has resulted in a 7% drop in stock price @xcite . we hypothesize that there may be other, uncovered factors influencing human judgments. in this paper, we examine a dataset of pairwise human judgments released by openai @xcite . much like a consumer weighing up multiple factors when deciding between two products, we aim to uncover the factors that evaluators consider when assessing two system outputs. we leverage the bradley-terry-luce model to analyze the strengths of these factors (bradley and @xcite , which has found applications in psychology, marketing, economics, and evaluation of natural language systems @xcite @xcite @xcite . it models pairwise comparisons, such as sports matches or political elections, by assuming each factor has an inherent strength, and the likelihood of one factor winning over another is a function of the difference between their strengths. we develop a protocol to identify characteristics of system outputs, ranging from output length and content coverage, to hallucinations of various sorts and the usage of complex words. our research sheds light on inherent preferences embedded in human judgments. ## problem formulation our dataset consists of a collection of n summary pairs, denoted by @xmath0 ## determinants of human preferences we explore a variety of factors that may influence human preferences. these factors are high-level, interpretable descriptors of the summaries and their original texts. our work distinguishes from previous studies using llms to evaluate the quality of hallucination & focus ## experiments in this section, we discuss various factors that affect human preferences. we quantitatively measure their degree of influence, how accurately these factors can be extracted from pairs of summaries, how factors correlate with one another, and their varying effects across summarization tasks and source domains. further, we evaluate how top-performing gpt models fare on this task. to ensure the replicability of our results, we set the temperature to 0 when using all gpt models. ## conclusion we aim to uncover the factors that influence human preferences. our research indicates that key factors are tied to users' information needs. users expect llms to understand their information-seeking intents, produce precise responses without irrelevant content or hallucinations, and generate moderatelength outputs of high linguistic quality. future research may also consider developing a balanced set of examples for human preference labeling to guide model behaviors more efficiently.