Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning (2024)

Abstract: Iterative preference learning, though yielding superior performances, requires online annotated preference labels. In this work, we study strategies to save annotation budgets while achieving competitive or even better performances for iterative preference learning. Built on intuitions from active learning, we empirically show that annotating those response pairs with small margins is generally better than large or random . Besides, experiments under the multi-iteration scenario suggest allocating more annotation budgets in the earlier iterations rather than later ones.

## introduction large language models (llms) @xcite have shown remarkable capabilities to understand and generate human languages, supporting applications such as question answering, coding, and psychological counseling. one of the keys to such success is to align llms with human-desired behaviors through preference learning. this is accomplished by annotating preference datasets and employing preference learning methods such as proximal policy optimization @xcite and direct preference optimization @xcite . to continuously improve llms' capability, recent work underlines the significance of iterative preference learning, which repetitively interleaves between training the model and collecting online preference annotations. for example, llama-2 and claude series benefited from iterative rlhf training on human preference annotations that were collected in batches on a weekly basis @xcite ; while multiple papers also reported that iterative dpo brings clear performance gains @xcite @xcite @xcite . despite their success, the process of collecting and annotating such online preference datasets is both time-consuming and costly. these methods normally sample multiple responses per instruction on a large new collection of instructions and simply annotate all the responses. the best and the worst responses are selected to formulate a pair per instruction to build a training corpus for the next iteration @xcite @xcite . this leads us to ask whether there exist alternative annotation strategies that are more cost-efficient. besides, existing methods normally allocate annotation budgets evenly across multiple iterations, but it remains unknown whether the model benefits from training on more instances in earlier or later iterations. all in all, we are interested in a research question: how to make better use of limited annotation budgets to aid online iterative preference learning. in this paper, we address this question by conducting a systematic study on iterative dpo based on llama-3-8b (ai@meta, 2024). we study the implicit reward margin as an informative indicator, which serves a key role in dpo and other direct preference learning methods ( §3). such a choice is supported by the formulation of dpo, in which the reward margin roughly represents prediction uncertainty from the discriminative perspective or distribution shift from the generative perspective. we consider two levels of granularity, namely the instance level and the corpus level, to rank reward margins from a comparative view. instance-level selection aims to find a worth-annotating pair of responses from n (n -1) 2 pairs if n responses are sampled per instruction; while corpus-level selection considers filtering out those trios that do little help for alignment tuning from a large set of trios each consisting of an instruction and a pair of responses. we conduct experiments in the single-iteration case, in which the policy llm goes through one round of online training after being initially trained on an offline dataset. we find that the smallestmargin subset always works better than the largestand random-subsets, on either the instance level or the corpus level. upon further checking the ranking accuracy and kl-divergences on each selected subset, we show that our assumptions regarding uncertainty and distribution shift are partly supported by our findings. we then generalize the winning strategy, always-smallest, to the multiiteration case ( §4). experimental results demonstrate that the always-smallest strategy yields continuous and significant improvements over multiple iterations; while the always-random strategy sees little to no gains upon training on more iterations. after that, we explore three strategies, namely increase, constant, and decrease, to allocate annotation budgets to multiple iterations. empirical results suggest it is better to adopt decrease and avoid increase. ## preliminaries in this section, we give a brief review of direct preference learning methods ( §2.1) and iterative dpo ( §2.2). then we present the preliminary setup of this work ( §3.2). ## margin-based selection within one iteration we begin by analyzing the data selection strategies within a single iteration. existing methods @xcite @xcite @xcite annotate all generated instances for the next iteration. we question whether other strategies could achieve better performance with the same amount of annotation budgets. we thus explore a simple yet intuitive metric for data selection, the reward margin between the chosen and the rejected responses. the reward margin ρ serves as the key component of the dpo loss function † : there exist multiple interpretations for ρ. the most straightforward one is to regard ρ as the reward margin between y w and y l as predicted by a pairwise rm. however, ρ is much more intriguing than that because it directly models the output logits of * such a definition is only valid under the case of reward margins so that the partition term is canceled. instead of staring at a single margin value that provides little insight for data selection, we consider ranking a set of reward margins. we assume the highest-or lowest-ranking subset might be of use to find the worth-annotating instances. we do not have a formal theory to support this assumption, but there are some intuitions from two points of view, i.e., discriminative and generative, associated with the mixed nature of dpo that optimizes a pairwise discriminative function that is built upon the output logits of generative llms. we will show empirical evidence to support these intuitions in §3.5. uncertainty upon regarding the policy model as a pairwise discriminative model, reward margins reflect the model's confidence in the prediction, so the most or least confident instances might be of extra use @xcite . specifically, @xcite showed that the calibration curve of a preference model roughly matches the logistic function, @xmath0, for models ranging from 10 8 to 10 10 parameters, demonstrating that reward margin is a good proxy of uncertainty. distribution shift as shown in eq ( 5) , reward margins represent the difference between the log ratios of the policy and the reference model; such a difference might correlate with the degree of generative distribution shift from the reference model to the policy. for example, if y w and y l lie in a similar distribution to that of the dataset where π θ was trained on, there should be a clear gap between the two log ratios since π θ has been trained to yield a higher generative probability for y w than y l while π ref has not. on the contrary, y w and y l may have not been effectively learned by π θ if the two log ratios were almost canceled and the margin is small, in which case π θ and π ref show similar generative behaviors to distinguish y w from y l given the above assumption that it might be useful to rank reward margins, the question then becomes how to define a good set of preference pairs on which we can get an informative ranking. in the next section, we will discuss two levels of granularity to define potentially useful sets for ranking. ## training for multiple iterations given our findings in the single-iteration case in §3, we would like to know how well the winning strategies generalize to multiple iterations (q4.1). besides, recent work reported continuous improvements with multiple iterations for iterative dpo @xcite . they all adopted an evenly distributed strategy to allocate annotation budgets across iterations. a natural question is, whether the model benefits from training on more instances in earlier or later iterations, rather than always training on the same amount (q4.2). the background experimental setup fol-lows that of §3.2. ## conclusion in this work, we investigated strategies to make better use of limited annotation budgets for iterative preference learning. through extensive experiments, we found that it is better to select the response pairs with smaller predicted reward margins and to allocate more annotation budgets in earlier iterations. we hope our findings could benefit the community to obtain better models with limited resources.