Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback (2024)

Abstract: Large language models (LLMs) fine-tuned with alignment techniques, such as reinforcement learning from human feedback, have been instrumental in developing some of the most capable AI systems to date. Despite their success, existing methods typically rely on simple binary labels, such as those indicating preferred outputs in pairwise preferences, which fail to capture the subtle differences in relative quality between pairs. To address this limitation, we introduce an approach called Margin Matching Preference Optimization (MMPO), which incorporates relative quality margins into optimization, leading to improved LLM policies and reward models. Specifically, given quality margins in pairwise preferences, we design soft target probabilities based on the Bradley-Terry model, which are then used to train models with the standard cross-entropy objective. Experiments with both human and AI feedback data demonstrate that MMPO consistently outperforms baseline methods, often by a substantial margin, on popular benchmarks including MT-bench and RewardBench. Notably, the 7B model trained with MMPO achieves state-of-the-art performance on RewardBench as of June 2024, outperforming other models of the same scale. Our analysis also shows that MMPO is more robust to overfitting, leading to better-calibrated models.

## introduction large language models (llms) trained on internet-scale data have demonstrated remarkable instruction-following and generalization capabilities, leading to their widespread adoption across natural language processing (nlp) tasks @xcite @xcite . pre-trained on a large, general corpus of text, llms acquire broad knowledge about the world with language understanding * equal contribution and reasoning abilities @xcite @xcite . to adapt a pre-trained llm to a downstream task, the model is typically fine-tuned on demonstrations of the desired output for the task. however, providing high-quality demonstrations is often more costly than simply evaluating model outputs. to address this, reinforcement learning from human feedback @xcite @xcite and ai feedback (rlaif; @xcite ) utilize feedback on diverse outputs to optimize llms, aligning the responses more closely with human intent. rlhf has been instrumental in developing some of the most capable ai systems to date @xcite @xcite . feedback-based alignment leverage human or ai feedback data to fine-tune generative models to better align their outputs with human intent. reward-based methods such as rlhf use feedback to learn a surrogate reward function, which is subsequently used in fine-tuning models via, e.g., reinforcement learning (rl). in contrast, rewardfree methods, such as direct preference optimization @xcite , bypass explicit reward modeling and directly fine-tune models using the data. feedback is commonly in the form of pairwise preferences, where responses to a given input are compared in pairs (e.g., y 1 ≻ y 2 for input x), and labels indicating the preferred one of the two are collected. more recent methods, such as kahneman-tversky optimization (kto; @xcite , also utilize simpler binary feedback indicating whether or not a response to an input is desirable. while these alignment methods have shown to be more effective than applying sft alone, existing methods rely only on binary labels indicating either the preferred output in a pair or the desirability of a single output, forgoing the opportunity to incorporate more granular feedback signals into learning. in this work, we introduce margin matching preference optimization (mmpo), a simple yet effective generalization of common alignment methods that integrates granular feedback signals into optimization, enabling models to capture subtle preferences in the feedback data (see figure 1 ). such granular feedback can come from human annotators providing detailed ratings, such as likert scale scores, or from ai models, which are increasingly used for automatic evaluation of model responses. the mmpo objective utilizes per-sample target preference probabilities, designed based on the quality margin between output pairs. since reward modeling and dpo, as well as similar methods based on pairwise preferences, use equivalent cross-entropy loss, the approach extends naturally to both. by incorporating target probabilities that reflect the quality margin of each output pair, models are trained to better account for the specifics of each feedback sample. for instance, when the quality margin is large, models trained with mmpo would assign significantly higher scores to preferred outputs, while assigning similar scores when the margin is narrow. the idea easily extends to methods that rely on alternative forms of feedback. our main contributions are as follows: ## preliminaries model alignment using feedback generally involves 1) supervised fine-tuning (sft) of pre-trained models and 2) aligning the models based on human or ai feedback data. in the sft phase, a pretrained llm is fine-tuned using supervised learning on task-specific demonstrations, such as humanwritten summaries in the case of a summarization task. in the alignment phase, the model is further fine-tuned to generate outputs that align with the preferences reflected in the feedback data. below, we review several popular approaches. rlhf. rlhf methods are reward-based approaches that first learn a reward function from feedback data, which is then used to provide training signals to the language model in rl finetuning. given a dataset d of pairwise preferences (x, y w , y l ), where x is an input and y w and y l are the preferred and dispreferred outputs, respectively, the reward function is trained to assign a higher score to y w than to y l . specifically, we model human preference probabilities using the bradley-terry model @xcite , which defines the probability as the sigmoid function applied to the difference in rewards given by the reward function r ϕ : ## margin matching preference optimization regardless of whether the feedback consists of pairwise preferences or indicators of desirability, the feedback label is generally binary. for pairwise preferences, we collect a binary label that identifies the preferred output from a pair. in the case of binary feedback, the label indicates whether or not an output is desirable for a given input. however, more detailed feedback is often available, particularly with the increasing use of llms and other ai models as annotators @xcite . motivated by this observation, we propose a simple generalization of common optimization objectives used in feedback learning methods, which results in improved reward models and language model policies. the main idea is to design per-sample target preference probabilities p(y w ≻ y l | x) for methods such as dpo that rely on pairwise preferences, and to adjust the loss weight for each (x, y) based on the desirability of y for x in methods such as kto that are based on binary feedback. ## experiments in this section, we evaluate mmpo by assessing the performance of the language model policies and reward models trained using the method on popular benchmarks. we utilize both human and ai feedback data, presenting benchmark performance along with an analysis of model calibration and the method's robustness to overfitting. ## related work fine-tuning pre-trained llms on task-specific data has become a standard approach to solving a range of nlp tasks. this adaptation typically involves supervised fine-tuning on task demonstrations, followed by aligning the models based on feedback from human annotators or ai models on various outputs. reward-based methods train a reward function on feedback data, which is then used, e.g., for rl-based fine-tuning @xcite . in contrast, reward-free methods bypass reward modeling to directly align models with preference data @xcite . feedback data is commonly in the form of pairwise preferences, where responses are compared in pairs, and binary labels indicating the preferred responses are collected. while this binary comparison is straightforward for human annotators, more fine-grained ratings are also often used @xcite . moreover, the growing use of ai models for evaluating outputs @xcite provides access to more granular feedback, allowing the use of these additional signals in alignment. for example, in developing the llama 2 models, human preference data was collected, with ratings that can be translated into a four-point scale @xcite . a fixed margin based on this rating is subtracted from the reward difference in the loss function, enhancing the accuracy of the reward model for evaluating response helpfulness @xcite . existing alignment methods, however, typically assume that the true preference probability is 1, i.e., the preferred response is always considered better than the dispreferred response. this limits incorporating such granular feedback signals into learning and increases the risk of overfitting @xcite . the issue of overfitting feedback data has been analyzed in the context of @xcite , but it is also relevant to reward modeling, as it shares the same formulation. label smoothing, a simple regularization technique in which a small constant is subtracted from the target probability, has been applied to dpo to derive a variant known as conservative dpo @xcite . while this variant has shown greater robustness against overfitting, it still fails to distinguish preferences with varying quality margins. identity preference optimization @xcite is a generalization of dpo that has been shown to address the overfitting issue, but it shares the same limitation of ignoring per-sample signals. several methods have also been proposed to address specific aspects of overfitting-related issues, such as generating lengthier responses post fine-tuning @xcite , by incorporating a constraint-based regularization term into the dpo loss. while these methods have been shown to mitigate specific issues, they still have the limitation of using hard targets. in contrast, our approach provides a more general framework by leveraging soft targets derived from relative quality differences, which can be designed based on factors such as response length, helpfulness, harmlessness, etc. moreover, our method can be easily integrated with constraint-based regularization techniques that modify the logits in the binary classification loss. ## conclusion in this work, we introduce margin matching preference optimization (mmpo), a simple generalization of common alignment methods that leverages granular feedback signals to enhance model optimization. our experiments with state-of-the-art open models on both human and ai feedback data demonstrate that mmpo leads to reward models and language model policies that outperform baselines on popular benchmarks in terms of both the quality of generated responses and their effectiveness as reward models. additionally, our analysis shows that mmpo is more robust to overfitting preference data, resulting in well-calibrated models that can better generalize to types of prompts unseen during fine-tuning.