The Paradox of Preference: A Study on LLM Alignment Algorithms and Data Acquisition Methods (2024)

Abstract: This research investigates the impact of preference annotation acquisition methods on the performance of LLM alignment algorithms, including Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), and Conservative DPO (cDPO), compared to Supervised Fine-Tuning (SFT) in NLP tasks. We analyze the influence of LLM and human-based preferences on algorithm performance, considering data volume and quality. Additionally, we assess DPO’s vulnerability to overfitting and IPO’s resilience against it, addressing four main research questions. Using the GAIR dataset and Zephyr-7b as the SFT model, we reveal unexpected negative outcomes. Specifically, DPO trained on LLM preferences outperforms human preferences, contrary to expectations. Moreover, there’s no correlation between preference data volume or quality and algorithm performance. Contrary to expectations, DPO shows no overfitting in both human and LLM preference datasets. Surprisingly, cDPO doesn’t fare better than DPO under flip noise. Our findings highlight the complexities of preference annotation methods and underscore the importance of scrutinizing negative results in NLP algorithm research.

## introduction large language models (llms) have proven their capacity to amass broad knowledge by simply maximizing the likelihood of human-written text but this objective isn't sufficient to generate responses that are safe, helpful and aligned with human preferences. methods based on reinforcement learning with human feedback (rlhf), including proximal policy optimization (ppo) @xcite , aim to align llms with human preferences, a theme also explored in other papers @xcite @xcite * work done during internship at observe.ai @xcite . direct preference optimization (dpo) @xcite was later shown to train policies in a single stage, treating it as a classification task using human preference data. it's favored over ppo for its ability to handle reward translation issues well and consistently achieve high rewards across different levels of kl divergence in generated text. due to the expensive nature of collecting human annotations, llm preferences serve as a substitute for human preferences in generating synthetic datasets (chiang and @xcite . reinforcement learning from ai feedback (rlaif) @xcite provides a promising alternative by leveraging a powerful off-the-shelf llm to generate preferences for large-scale model training. the use of llm preferences in dataset creation @xcite has shown comparable performance between rlaif and rlhf across various tasks, with performance degradation attributed to dataset quality issues, as evidenced by human-agreement scores. conservative dpo (cdpo) 1 addresses these challenges by adopting a conservative target distribution, minimizing error probability, and deriving a loss function to ensure alignment between model preferences and observed preferences. the scarcity of diverse preference datasets poses a challenge for rlhf and feedback learning research. ul-trafeedback @xcite addresses this challenge by providing an extensive, high-quality, and diversified preference dataset. while widely adopted in preference optimization, dpo is susceptible to overfitting as observed by @xcite in the initial epoch of zephyr-7b dpo training, but noted improved performance with further epochs. the ipo paper @xcite discovered that rlhf and dpo are prone to overfitting due to relying on the assumption that pairwise preferences can replace elo-scores through bradley-terry modeling. to mitigate this, ipo introduces a regularizing term controlling log-likelihood ratios to address overfitting to the preference dataset. a literature gap exists in exploring how training data volume influences llm alignment algorithms, dpo and ipo. empirical evidence is lacking on ipo's ability to counter dpo's overfitting, and studies on data quality's impact on dpo, and cdpo's effectiveness in addressing it, are scarce. it's essential to investigate the influence of preference annotation methods (llm vs. human preferences) on these factors and the performance of llm alignment algorithms, including dpo, cdpo, and ipo, given the increasing use of llm preferences. in this work, we investigate how the method of preference annotation acquisition affects the critical performance factors influencing the effectiveness of llm alignment algorithms and seek to address the following research questions: we demonstrate unexpected superiority of llm trained with dpo on llm preferences over human preferences. performance shows no correlation with data volume or quality. dpo doesn't exhibit overfitting issues, while cdpo doesn't improve under noise. our findings highlight challenges in preference annotation and aligning llms. ## implementation details we choose zephyr-7b as our sft model and gair @xcite as our preference dataset, contain-ing both human and llm preferences, for our experiments. mt bench @xcite is used to evaluate our models while gpt4-turbo is chosen as the llm for obtaining synthetic preferences. further details on our choices are provided in section a.1, including hyperparameter specifics. ## results and analysis in this section, we provide a comprehensive analysis of the performance evaluation results, shedding light on the key observations made during our study. ## conclusion we analyze the influence of data quantity on dpo and ipo, utilizing llm preferences and human preferences. surprisingly, there's no linear correlation between data quantity and performance. similarly, the impact of data quality on dpo and cdpo, using both llm preferences and human preferences, also lacks a linear trend with performance. contrary to expectations, dpo trained on llm preferences outperforms its human-trained counterpart. additionally, ipo fails to outperform dpo across various data volumes, while cdpo struggles to address induced flip noise in preferences. interestingly, dpo shows no signs of overfitting when trained on both llm and human preference datasets. these findings prompt further research to enhance the resilience and effectiveness of llm alignment algorithms in preference modeling.