Dissecting Human and LLM Preferences (2024)

Abstract: As a relative quality comparison of model responses, human and Large Language Model (LLM) preferences serve as common alignment goals in model fine-tuning and criteria in evaluation. Yet, these preferences merely reflect broad tendencies, resulting in less explainable and controllable models with potential safety risks. In this work, we dissect the preferences of human and 32 different LLMs to understand their quantitative composition, using annotations from real-world user-model conversations for a fine-grained, scenario-wise analysis. We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. On the contrary, advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. Additionally, LLMs of similar sizes tend to exhibit similar preferences, regardless of their training methods, and fine-tuning for alignment does not significantly alter the preferences of pretrained-only LLMs. Finally, we show that preference-based evaluation can be intentionally manipulated. In both training-free and training-based settings, aligning a model with the preferences of judges boosts scores, while injecting the least preferred properties lowers them. This results in notable score shifts: up to 0.59 on MT-Bench (1-10 scale) and 31.94 on AlpacaEval 2.0 (0-100 scale), highlighting the significant impact of this strategic adaptation. We have made all resources of this project publicly available.

## introduction human and llm preferences have played a crucial role in the development pipeline of recent advanced language models. preference-based training, such as reinforcement learning from human/ai feedback (rlhf/rlaif) @xcite and direct preference optimization (dpo) @xcite , are widely used to fine-tune models to align with more practical needs. on the other hand, the preferences of human and llms, in the form of llm-as-a-judge or elo ratings, have become the de facto judging criteria for assessing the quality of model outputs @xcite as the tasks are becoming increasingly diverse and complex. unlike the widely applied preference-based methods above, the preferences themselves lack thorough research. in most cases, they are only binary labels indicating which response is preferred as a vague form of expression, and we are unable to understand the preferences in an explainable and quantitative way. as a result, optimizing models towards such goals inevitably leads to certain issues @xcite . these include the trained models engaging in over-optimization @xcite and reward hacking @xcite , manifesting in undesired ways such as producing overly verbose answers @xcite or demonstrating sycophancy @xcite , which hinder the building of more reliable ai systems. in this work, we build a systematic framework to dissect the overall preferences into a quantitative combination of multiple clearly defined properties. to pursue understanding in realistic settings, we sample real-world user conversations with a balanced distribution of different scenarios from chatbot arena conversations @xcite , where each sample is a pair of model responses to a query. we adopt an elaborate yet automated pipeline to annotate the data with regard to our pre-defined properties (e.g., harmless or admit limits). based on the annotations, we determine how a pair of responses differ from each other on all properties. these distinctions are then used to fit prefer dislike neutral figure 1 : the preference dissection of human, gpt-4-turbo and llama-2-70b-chat on communication scenario (including chitchat and value judgment), and we highlight the most preferred property for each in its corresponding color: lengthy for human, no severe errors for gpt-4-turbo, and contain rich info for llama-2-70b-chat. the value is the probability of a response being preferred in a pair when it satisfies one property better than the other response, holding all else equal. this can be interpreted as how much human or an llm favor a certain property. values above and below the 50% line indicate a preference or dislike, respectively. bayesian logistic regression models, which help us quantitatively decompose preferences based on different properties by examining their weights. leveraging the above framework, we analyze the human preferences of real users and the preferences of numerous llms we collect. the analysis is conducted separately on different scenarios to avoid the mixing of preferences and achieve clearer conclusions (see figure 1 for an example). we summarize the key findings as follows: finally, we reveal that benchmarks with llmas-a-judge are easy to manipulate ( §4.3). our experiments on alpacaeval 2.0 @xcite and mt-bench @xcite show that aligning models with the judges' preferences increases scores, whereas diverge from these preferences leads to lower scores. this is achievable across both training-free and training-based methods, with score variations up to 0.59 on mt-bench (1-10 scale) and 31.94 on alpacaeval 2.0 (0-100 scale). the manipulation highlights the urgent need for more robust benchmarks and further underscores the importance of understanding preference. ## related work the application of preferences in training, preference data through pairwise comparisons is used to build the reward model in reinforcement learning from human/ai feedback (rlhf/rlaif) @xcite @xcite or as the learning target in direct preference optimization (dpo) @xcite @xcite . in evaluation, comparing model outputs to references @xcite @xcite , directly rating individual responses @xcite or using elo ratings from human votes @xcite have become common ways to assess the aligned models. ## analysis and application as the most common sources of preference data, we thereby analyze two specific cases: human and one of the most advanced llms, gpt-4-turbo. ## conclusion in this work, we conduct a thorough analysis to dissect how the preferences of human and llms can be quantitatively decomposed into different properties. we find that humans prefer responses that can directly address their queries but are less sensitive to errors in responses, while advanced llms like gpt-4-turbo emphasize more on correctness, clarity, and harmlessness. we then find that model size can be a distinguishing factor in the preferences where llms of similar sizes share similar preferences, and fine-tuning for alignment does not bring about significant change to llm preferences. as an application of our analysis, we also show the results of current benchmarks with llm-as-a-judge can be intentionally manipulated, indicating the vulnerability of preference-based evaluation. last but not least, we publicly release all the collected resources to facilitate future research.