DORM : Preference Data Weights Optimization for Reward Modeling in LLM Alignment (2025)

Abstract: Aligning large language models (LLMs) with human preferences relies heavily on high-quality reward models. However, existing approaches struggle with two critical challenges: noisy preference labels and the varying importance of preference samples. We introduce DORM, a method that enhances reward modeling by learning to dynamically weigh preference data.DORM initializes data importance using a combination of model uncertainty and prediction disagreement, then iteratively refines them via bilevel optimization to maximize validation performance. Using only 50k samples, DORM trains a 12B reward model that achieves 90.5% accuracy on RewardBench, matching the performance of models trained on significantly larger datasets. Furthermore, downstream alignment tasks show that fine-tuned LLMs with DORM achieve a 61.2% win rate against baseline methods, highlighting its data efficiency and generalizability.

## introduction aligning large language models (llms) with human preferences is crucial to ensure llms adhere to human values. recent advances in llm alignment have primarily focused on algorithmic innovations, ranging from reinforcement learning from human feedback (rlhf) @xcite to alternative preference learning approaches like sequence likelihood calibration (slic) @xcite and direct preference optimization (dpo) @xcite . although such algorithmic advancements have enhanced llm alignment, the quality of preference data used for reward modeling has received limited attention. these existing approaches face three key limitations: (1) they often rely on heuristics or expensive oracle feedback, (2) they treat ambiguous preferences as unreliable samples rather than potential learning opportunities, and (3) they fail to adaptively adjust data importance based on the reward modeling progress. these limitations raise a critical research question: can we develop a learning-based approach to automatically estimate the importance of preference data, thereby enhancing reward modeling? we address this challenge by introducing dorm (preference data weights optimization for reward modeling) -a two-stage approach that combines preference data quality estimation with adaptive data weighting. the key intuition of the first stage is that data points where the model is uncertain are often informative and should be emphasized, whereas mislabeled or unreliable samples should be down-weighted. to this end, we estimate preference data weights by integrating model uncertainty and prediction disagreement. specifically, we employ approximate bayesian inference techniques for uncertainty estimation @xcite , while measuring disagreement by the discrepancy between predictions and labels. this procedure prioritizes informative and reliable data and mitigates the impact of unreliable data. the second stage further refines data weighting by dynamically adjusting weights through a bilevel optimization framework. this approach maximizes validation performance while using initial uncertainty-based data weights as regularization. the framework consists of two levels: (1) the upper-level optimization adjusts data weights to minimize validation loss, guided by initial weight estimates; (2) the lower-level optimization trains a reward model using the weights determined by the upper-level problem. this learning process automatically identifies which data points are most useful for improving validation performance while mitigating overfitting through uncertainty-based regularization. to evaluate our method, we conduct experiments on both the reward model and policy model levels. our 12b reward model, trained with only 50k preference data samples, achieves 90.5% overall performance on reward-bench, matching the performance of similarsized models trained with significantly more data. analysis of data weight tracking reveals that high-quality datasets are progressively assigned greater weights, while noisy samples are gradually down-weighted. furthermore, the policy model aligned using our reward model shows 61.2% win rates compared to using the baseline reward model. our contributions can be summarized as follows: 1) preference data quality estimation: we introduce a method for estimating preference data quality by combining epistemic uncertainty and disagreement measures. 2) bilevel optimization framework: we formulate learning to weigh preference data as a bilevel optimization problem, enabling datadriven refinement of data weights while incorporating initial quality estimates. 3) data-efficient reward model training: our approach achieves 90.5% accuracy on reward-bench, with 10 -40× less preference data compared to baselines. ## preliminaries multi-objective reward modeling. let x and y denote the space of prompts and responses, respectively. unlike traditional reward models that rely solely on pairwise preferences @xcite @xcite , multi-objective reward modeling leverages finegrained ratings across multiple reward attributes @xcite to capture richer preference signals. in this setup, human feedback is collected in the form of structured ratings, where each response @xmath0, with each dimension corresponding to a different reward attribute. given a dataset @xmath1, y i , r i )} |d| @xmath2, where x i represents the input prompt, y i is the generated response, and r i is the associated multi-dimensional rating, the objective is to learn a reward model r θ that predicts these ratings accurately. specifically, given an input pair (x, y), we concatenate the prompt and response as x⊕y and pass it through a pre-trained decoder-only llm m ψ , extracting a d-dimensional representation from the final decoder layer. a linear regression head @xmath3, which is optimized using the following regression loss: projection for scalar metrics. since the reward model operates in a multi-objective setting, where each response y is evaluated across m different reward attributes, it is often necessary to project the reward vector @xmath4. to achieve this, we introduce a projection vector λ ∈ r m , which allows us to map the multi-dimensional reward vector onto a single scalar value by @xmath5. ## preference data weight optimization with estimated quality in this section, we formulate the weight optimization for preference data as a bilevel optimization problem. our approach operates in two stages. in the first stage, we estimate initial data weights based on epistemic uncertainty @xcite and a disagreement function. these measures help identify more reliable and informative data points, guiding the model towards higher-quality examples. the second stage employs a bilevel optimization framework @xcite . the lower level optimizes reward model parameters using weighted preference data, while the upper level refines these weights to minimize loss on a high-quality validation set. critically, we incorporate the initial weights as regularization terms in the upper-level problem, anchoring the optimization to the prior quality estimates and stabilizing the process. we present our method overview in figure 1 . ## conclusions in this study, we presented dorm to tackle the critical issue of preference data quality estimation in aligning llms. by integrating epistemic uncertainty with a disagreement measure, we developed a method to assess the informativeness and reliability of each preference data point. by utilizing these quality estimates as prior weights and refining them through a bilevel optimization framework, we balance prior knowledge with data-driven insights and enhance the robustness of reward models to handle diverse preference data. dorm achieves high performance with significantly less data, leading to more robust reward models and better-aligned policy models. this work highlights the importance of data quality in model alignment and provides a promising avenue for developing more reliable and human-aligned language models. limitations 6.1 dataset-level weight assignment for computational efficiency, our current implementation assigns weights at the dataset level. while this simplification maintains good performance and computational efficiency, it may not fully capture the fine-grained quality variations within each dataset. future work could explore hierarchical weight optimization strategies for finer-grained assignments.