Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment (2025)

Abstract: Large Language Models (LLMs) are often aligned using contrastive alignment objectives and preference pair datasets. The interaction between model, paired data, and objective makes alignment a complicated procedure, sometimes producing subpar results. We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training. Based on these insights, we introduce Contrastive Learning from AI Revisions (CLAIR), a data-creation method which leads to more contrastive preference pairs, and Anchored Preference Optimization (APO), a controllable and more stable alignment objective. We align Llama-3-8B-Instruct using various comparable datasets and alignment objectives and measure MixEval-Hard scores, which correlate highly with human judgments. The CLAIR preferences lead to the strongest performance out of all datasets, and APO consistently outperforms less controllable objectives. Our best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, closing the gap with GPT4-turbo by 45%. Our code and datasets are available.

## introduction aligning language models with preferences is a critical component in llm development, significantly enhancing model capabilities, safety, and adherence to human values @xcite @xcite . these preferences can be expressed through preference pairs (output y l ≺ y w for input x), which of-fer a richer signal than individual outputs and enable more expressive learning objectives. recently, contrastive learning objectives have made alignment more accessible @xcite . despite these advantages, alignment outcomes can be suboptimal @xcite @xcite . in this paper, we reason through the nature of alignment, focusing on (i) the preference signal expressed by the data and (ii) the training dynamics of contrastive objectives. we find that across both these axes, conventional alignment methods are underspecified. to solve this, we argue that (i) preference data should be minimally contrastive, and (ii) alignment objectives should account for distinct alignment situations (see figure 1 ). this sheds light on suboptimal alignment outcomes. for example, we show in section 5 how a model aligned using high-quality outputs can actually degrade if the pairs differ in multiple uncontrolled aspects. these insights lead to two new contributions. first, we introduce contrastive learning from ai revisions (clair), a method for creating preference pairs which minimally revises one output to express a preference. the pairs created by clair result in a more precise learning signal, as opposed to conventional methods which use a judge to select a preferred response. second, we introduce anchored preference optimization (apo), a family of contrastive objectives which explicitly account for distinct relationships between model and data during alignment. the tailored training dynamics of apo results in more performant alignment compared to conventional objectives. in order to study the role of both (i) minimally contrastive preference data, and (ii) distinct alignment training dynamics, we individually align a model across four comparable preference datasets using five alignment objectives. one dataset is created through our clair method. we compare this with two conventional judge-based datasets (reinforcement learning from ai feedback; @xcite . finally, we consider an ablated version of clair created to directly assess the impact of contrastiveness. we consider five distinct alignment objectives: dpo @xcite , kto @xcite , continued supervised fine-tuning on the preferred answer, and two variants of our proposed apo. we measure mixeval-hard accuracy @xcite and length-controlled alpacaeval scores @xcite for each model, both benchmarks correlate highly with model rankings produced by humans @xcite . we align llama-3-8b-instruct @xcite and use gpt4-turbo @xcite for preference judgments / revisions. we find that our strongest model, aligned on 32k clair preferences with apo, improves llama-3-8b-instruct performance by 7.65% on mixeval-hard, closing the performance gap with gpt4-turbo by 45%. our analysis indicates that the contrastiveness of clair preferences is the major driver of performance. across every alignment datasets considered, apo objectives achieve the best performance. in our analysis, we outline how to select the best apo variant given a target model and preference dataset. finally, we explore recent alignment efforts and discuss how they relate to clair and apo. ## underspecification in alignment the alignment procedure creates complex interactions between the target model, the preference dataset, and the alignment objective. the present section reflects on failure cases of all alignment efforts which start from preferences. the section discussed data and objective respectively. given a collection of prompts x, a preference dataset is a set of triples (x, y w , y l ), where y w and y l are, respectively, a winning (more preferred) and losing (less preferred) response to prompt x. the preference signal in such a dataset is essentially expressed by the difference between winning and losing outputs, illustrated in figure 1a . however, paired outputs can differ in many aspects, some of which can be spurious and thus irrelevant to the preference. these spurious differences can generally create a challenging credit assignment problem. outputs which are minimally contrastive differ along fewer axes, resulting in less spurious differences. thus, if preference pairs produce a clearer minimal contrast, the alignment learning signal becomes clearer. existing preference datasets vary meaningfully in their contrastiveness. for example, in the stanford human preferences dataset @xcite , two outputs in a pair are simply responses to the same reddit post, and thus they are not guaranteed to be especially comparable. an ideal preference dataset would consist of a very controlled difference between either example. this insight leads us to clair (section 3). preference triples only specify that one output is better than another. this creates ambiguity, since it is not known if the more preferred answer was actually good. to see how this can impact alignment, suppose we have a dataset of triples where y w tends to score 8/10 on some quality scale and y l tends to score 6/10. a target model that generally scores 9/10 may become worse if the likelihood of y w would increase during training, as illustrated in figure 1b . therefore, alignment training needs to be aware of how desirable any individual answer is, regardless of its preference relationship. to take a salient example, ≈80% of winning outputs in ultra-feedback @xcite are generated by a less performant model than llama-3-8b-instruct (as measured by chatbot arena elo; @xcite . naively aligning revision of this answer. the differences between answer and revision are highlighted. the revision generally follows the same outline as the answer but improves it where possible. for example, the revision correctly alters the count of parisian restaurants from 2 to 3 in the second line of the answer. llama-3-8b-instruct on this dataset may thus worsen performance. examples like this one lead us to anchored preference optimization (apo; section 4). in summary, current alignment approaches are underspecified along two key axes: (i) preferences may be weakly expressed due to noncontrastive data and (ii) alignment objectives need to account for the model-data relation. in what follows, we set out to improve alignment across both axes. ## contrastive learning from revisions we now introduce contrastive learning from ai revisions (clair), a general procedure for creating minimally contrasting preference pairs. let m be the target model we will align. given a prompt x, we sample the losing output y l directly from the model. then, we use a reviser to minimally revise and improve y l , resulting in the winning output y w : in this work, we use a stronger llm to perform revisions, prompted to enhance the clarity, correctness, and engagement of the output (prompts and dataset details given in appendix a). figure 2 shows an example triple created using this method. the losing output was generated by llama-3-8b-instruct and revised by gpt4-turbo. the revision keeps most of the initial output intact, while improving details. recently, @xcite used human revisions in the development of the llama-3.1 model family, though their process seems oriented towards enhancing quality differences rather than creating minimal contrasts. clair differs markedly from more familiar approaches to collecting preference data. for example, in the on-policy judge paradigm (as used in reinforcement learning from ai feedback; @xcite , two generations are sampled from m (x), and a judge (often another llm) decides which is the winner and which is the loser: we use this approach as one of our baselines, with a prompt comparable to the revision prompt used by clair. additionally, we consider an off-policy judge version of (2) where the outputs are generated by models other than the target model: both the on-policy and off-policy judge approaches provide useful comparison points for clair. in addition, we evaluate a baseline that helps us understand the role of contrastiveness in particular. for clair, the reviser is generally a stronger model than the model we are aligning. this means that the winning examples y w are always generated by a stronger model. to decouple this factor from the contrastiveness induced by the revision process, we also evaluate a baseline that we call stronger preferred, where the stronger model provides the winning example for each pair without revision: for the alignment experiments reported in section 5, we created four preference datasets following (1)-(4). each dataset is created using the same 32k prompts uniformly sampled from ultrafeedback @xcite , a widely used preference dataset with prompts spanning a broad range of domains. we take the target model m to be llama-3-8b-instruct, one of the most competitive open source models available at the time of writing. we use gpt4-turbo to act as the judge, reviser, and stronger model when creating these datasets. for the off-policy judge dataset, we use already judged outputs available in ultrafeedback. approximately 80% of these winning outputs are generated by a model weaker than llama-3-8b-instruct (as measured by chatbot arena elo; @xcite . thus, this off-policy judge dataset generally contains lower quality outputs compared to the model. part of the goal of section 5 is to study the behavior of each of these datasets in the context of alignment efforts. however, one of the highlevel goals of clair is to generate examples that are minimally contrastive. we can assess this directly using some simple heuristics: the jaccard similarity (token intersection over union) between y w and y l and the single-character levenshtein edit distance between y w and y l . the dataset with better minimal contrasts should result in a higher jaccard similarity and a lower levenshtein distance. ## anchored preference optimization a preference triple (x, y w , y l ) expresses the belief that y w is a more preferred output than @xcite ) is a widely used and empirically successful alignment objective. the core stipulation of dpo is that the likelihood change of winning outputs during training needs to be greater than the likelihood change of losing outputs. this likelihood change for a prompt and output is denoted as the reward r θ (x, y), which captures the logratio of likelihoods between the model during training π θ (x | y) and the model before training, also called reference, here, β is a hyperparameter which scales this log-ratio. this leads to the following dpo objective: the dpo authors report that the gradient of this objective intuitively leads to an increased winning likelihood and decreased losing likelihood. however, this is only one possibility out of three distinct scenarios. alternatively, dpo can increase the winning likelihood more than it each gradient term is decomposed in a direction and magnitude factor. direction: either apo variant specifies explicitly if winning and losing likelihoods should increase or decrease during training. dpo only increases the likelihood difference, causing ambiguity with regard to the actual movement of these likelihoods during training. this explicit specification of direction is core to apo variants, and allows for a tighter fit between model and data during alignment. magnitude: each term in apo is scaled with a delta function. here, to help researchers navigate these interactions, we introduce anchored preference optimization (apo). in essence, apo is a family of alignment objectives which offer fine-grained control over each of the rewards, thus controlling the absolute increase or decrease in likelihood during training. in this paper, we focus in particular on variants that we call apo-zero and apo-down: apo-zero explicitly pushes for an increased likelihood of winning outputs and decreased likelihood of losing outputs during training. in contrast, apo-down decreases the likelihood of winning outputs and decreases the likelihood of losing outputs even more. if answers from the model are on average better than the winning outputs (y w ≺ π θ ), apo-down will intuitively be a better objective. if winning outputs are better than the model answers (y w π θ ), apo-zero will be better. figure 3 provides an interpretation of the gradients produced by both apo methods and compares these with dpo. one can define additional apo objectives. in general, any contrastive objective (i.e., greater reward for winning outputs) which specifies additional constraints on either reward to achieve a tighter link between model and data (e.g., winning rewards should be positive) can be seen as a form of anchored preference optimization. in section 6 we consider different alignment objectives and discuss how they relate to apo. one interesting variant of apo can be derived from the kahneman-tversky optimization (kto) objective of @xcite . as originally defined, kto does not operate on preference pairs, but rather requires only one unpaired answer and a label indicating if it was preferred or not; the goal of kto is to push the winning / losing reward above / below the kullback-leibler (kl) divergence between the model during training and the reference model. the apo perspective helps us see that there is a natural paired variant of kto in which the kl-divergence functions as the anchor: this kl term is non-negative, and thus the winning reward is pushed to be positive; the losing reward can still be either positive or negative. the kto authors report that kto leads to good alignment without an initial phase of supervised fine-tuning (sft) on the winning outputs, while dpo does benefit from this sft phase in their experiments. apo sheds new light on this finding: an increase in likelihood of winning outputs is already built into kto, whereas it is not guaranteed for dpo alone. however, this is only a desirable property of an alignment objective if the winning output quality is better than the target model's quality, as described in section 2. when aligning a strong model on preferences which contain generally lower quality outputs, a kto-style objective runs the risk of deteriorating the model. ## alignment experiments to study the effectiveness of clair and apo, we align llama-3-8b-instruct across the four comparable preference datasets described in section 3, created from 32k ultrafeedback prompts. we use gpt4-turbo to act as the judge, reviser, and stronger model when creating these datasets. for the off-policy judge dataset, we use the already judged outputs included in ultra-feedback. for every dataset, we align the model using the four different objectives described in section 4. additionally, we consider supervised fine-tuning (sft) on only the winning outputs as a baseline alignment objective. ## related work we now characterize relevant alignment efforts and outline how they relate to contrastive learning from ai revisions (clair) and anchored preference optimization (apo). reinforcement learning from human or ai feedback (rlhf / rlaif; @xcite @xcite is a technique used to align models with human preferences. fundamentally, these approaches first train a reward model using preference judgments and subsequently optimize a language model for this reward using reinforcement learning @xcite . to side-step the need for an explicit reward model, direct preference optimization (dpo; @xcite aligns an lm directly using a contrastive training objective. we articulated two core insights concerning (i) the role of contrastive preference data, and (ii) the need to anchor alignment depending on model and data. these insights translate to any alignment effort which uses comparative preferences. for example, a reward model trained on spurious preference signals may be a less accurate proxy for real rewards, contributing to problems such as reward overoptimization or hacking @xcite . for the remainder of this review, we first focus on contrastive alignment methods and their variants (of which @xcite provide a detailed overview). finally, we discuss related preference datasets and how they were created. changing the lm more / less: @xcite and @xcite recognize that preference pairs can vary. both works study how much more preferred the winning output is, and seek to incorporate this into the objective by changing the model more / less depending on this preference strength. using the difference in gold rewards as a substitute for preference strength, @xcite add an instance-level margin to the contrastive objective while @xcite scale the β parameter at a batch-level. other works also utilize a margin in the contrastive loss, but specify this as a static hyperparameter @xcite @xcite . these contributions complement our own; they focus on how much a model should change, whereas clair creates better learning signals and apo more fully specifies the intended training dynamics. ## future work in this work, we have presented two variants of the apo objective family. each method accounts for a distinct relationship between target model and preference pair during training. however, real world preference datasets may contain a wide range of different preference pairs, thus the dataset as a whole may not perfectly correspond with any single apo variant. to tackle this, a natural extension of apo could be to select the optimal apo variant at the preference pair level, instead of at the dataset level. heuristically, this could be achieved using an off-the-shelf reward model to score each preference pair before training. we used prompted llms to create our datasets through revisions or judgments. the distinction between a model and a system of models is arbitrary, and future work could improve clair's performance by using a system of models to produce a revision of higher quality instead. additionally, there is a natural trade-off between how much change a revision introduces and how much quality it adds. in many cases it can be challenging to minimally improve a given answer, it is not clear what level of revision would be optimal for alignment. this can be studied empirically by creating different versions of our clair dataset with increasingly intense revisions. ## conclusion alignment performance is significantly impacted by (i) the contrastiveness of the preference pairs and (ii) the relationship between target model and alignment data. we introduce contrastive learning from ai revisions (clair), a data-creation method which produces better contrasting preference pairs, and anchored preference optimization (apo), a family of alignment objectives with tailored training dynamics. our experiments aligning llama-3-8b-instruct show that clair preferences lead to the highest performance improvement out of four comparable preference datasets, and apo methods consistently outperform conventional alignment objectives. @xcite . mixeval-hard uses a gpt3.5-turbo model to rate if a response to a query agrees with a known gold-truth response.