Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora (2025)

Abstract: Inconsistent annotations in training corpora, particularly within preference learning datasets, pose challenges in developing advanced language models. These inconsistencies often arise from variability among annotators and inherent multi-dimensional nature of the preferences. To address these issues, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them. Our method enhances preference learning by automatically detecting and selecting consistent annotations. We validate the proposed approach through extensive instruction-following tasks, demonstrating performance improvements of up to 33% across various learning algorithms and proxy capabilities. This work offers a straightforward and reliable solution to address preference inconsistencies without relying on heuristics, serving as an initial step toward the development of more advanced preference learning methodologies. Code is available at https://github.com/Self-Curation/ .

## introduction aligning language models (lms) with human intent is crucial for reliable deployment as unhelpful or harmful responses can pose significant risks. preference learning methods -such as reinforcement learning from human feedback (rlhf) @xcite @xcite @xcite and its successor, direct preference optimization (dpo) @xcite -leverage preference datasets to achieve alignment that goes beyond what instruction tuning can offer. unlike instruction tuning, which relies on single-response supervision, preference datasets enable more explicit learning by comparing chosen and rejected responses to given instructions. due to their two-response structure, however, preference datasets are vulnerable not only to noise * equal contribution within individual responses but also to incorrect flipping of preference between responses. moreover, because these datasets must consider a wide range of criteria (e.g., helpfulness and harmlessness), the preference decision often requires navigating a more complex, multi-dimensional space, complicating the annotation process. this complexity is exacerbated by the diverse cultural or social backgrounds of annotators, leading to inconsistent preferences or unreliable annotations. during the collection of paired corpora for prompts, decisions can vary from unanimous to narrowly divided even within the same crowd @xcite . however, this variability is often excluded from the training dataset, introducing noise from inconsistent annotations or conflicting criteria, which can distort the target model's learning of desirable preference patterns. similar issues may arise when using large language models (llms) for annotation as they exhibit human-like patterns @xcite . @xcite observed that gpt-4 (ope-nai, 2023) can display inconsistent behavior during multi-step reasoning despite its impressive performance on various tasks. to mitigate these challenges arising from noisy data in the training dataset, manual data cleansing or adding additional annotations, such as the significance level of each preference data point, can be used @xcite . however, these approaches are costly and time-consuming. moreover, determining whether data is truly noisy can often be particularly challenging. among earlier literature on the dataset quality, @xcite categorize training data for language models into 'easy-to-learn', 'ambiguous' and 'hard-to-learn' and show that an appropriate mix of easy-to-learn and ambiguous data is essential for successful optimization. however, their study is largely confined to tasks requiring short answers and leaves the question of how to achieve the optimal balance unresolved. to address this limitation, we propose a method for selecting a well-balanced subset from the target preference dataset to enhance the training of generative lms. for more complex generative tasks, several prior studies have focused on data selection to enhance instruction tuning performance. whereas @xcite suggest that ambiguous tasks can enhance generalization in instruction tuning using a categorization similar to @xcite , @xcite demonstrate that selecting easyto-learn data over hard-to-learn segments is more beneficial for improving performance. they argue that data with low prediction probability may be too specific thus hard to learn. other related studies include @xcite ; @xcite ; @xcite , but these works mainly focus on data efficiency and are limited to instruction tuning. in contrast, our approach offers a new perspective on dataset quality by introducing a novel subset selection method that enhances the consistency of preference annotations for more effective preference learning. some previous approaches to preference learn-ing aim to build better reward functions by merging or weighting multiple reward models through a multi-objective learning framework @xcite @xcite . however, the underlying issues of dataset quality remain unaddressed. additionally, conservative dpo (cdpo) @xcite and robust dpo (rdpo) @xcite consider mislabeled or ambiguous data contained in the dataset unlike dpo. however, their approach assumes the error rate as a prior instead of providing a systematic way to determine it, which reduces its practicality. in this work, we propose simple yet effective methods to fully automate the curation of training corpora that increases preference consistency in the dataset. our self-curation method leverages proxy models trained on the given preference datasets using the bradley-terry (bt) model @xcite . through this training process, consistent preference patterns emerge, enabling the trained proxy to retrospectively evaluate the original annotations to assess their consistency. we assess preference consistency by identifying discrepancies between the proxy models' predictions and the given annotations, as illustrated in figure 1 . this inconsistent preference patterns are challenging for the target lm to comprehend @xcite , making such corpora detrimental to preference learning. we demonstrate that simply excluding these inconsistent data from training can significantly enhance preference learning such as dpo through extensive experiments. the key contributions of this work are summarized as follows: ## preference consistency noisy labels are prevalent in preference datasets often arising from vague annotation criteria and ambivalent data that induce conflicting preferences, making it difficult for annotators to articulate their choices. in some cases, paired responses differ only slightly or are nearly identical (figure 2 ), which may not provide a clear basis for the model to learn meaningful preferences. gpt-4 as a judge exhibits similar patterns to human annotators (bubeck et al., 2023), facing the same issues (figure 3 ). moreover, individual factors of annotators can lead to inconsistent judgments on the same data, potentially compromising the consistency of the dataset. ## self-curation based on the investigation in sec. 3, we propose a self-curation method for a preference dataset d p as described in algorithm 1. self-curation (1) trains a proxy r ϕ on d p , (2) estimates scores of d p paired instances using r ϕ , and (3) selects consistent preference data d c . after screening out preference discrepancies, the remaining subset d c becomes the self-curated dataset as illustrated in figure 1 . we clarify that we do not claim that evaluations by smaller models are more accurate than those by 2 ) in this section, we demonstrate that improving the dataset consistency with self-curation leads to higher alignment performance (sec. 4.1), extend our approach to other methods such as cdpo and rdpo to validate its effectiveness (sec. 4.2) and examine the impact of the proxy capability by using various base models for proxy training (sec. 4.3). ## conclusion in this work, we addressed the inherent challenges of preference inconsistency in aligning lms. we proposed a straightforward and reliable selfcuration method that leverages proxy models to identify and select data with consistent preferences. our approach significantly and consistently improves preference performance in an entirely automated fashion. its effectiveness, independent of proxy capability and preference learning methods, was validated across diverse instruction-following tasks, underscoring its potential to refine preference alignment in lms.