BPO : Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment (2025)

Abstract: Reinforcement Learning with Human Feedback (RLHF) is the key to the success of large language models (LLMs) in recent years. In this work, we first introduce the concepts of knowledge breadth and knowledge depth, which measure the comprehensiveness and depth of an LLM or knowledge source respectively. We reveal that the imbalance in the number of prompts and responses can lead to a potential disparity in breadth and depth learning within alignment tuning datasets by showing that even a simple uniform method for balancing the number of instructions and responses can lead to significant improvements. Building on this, we further propose Balanced Preference Optimization (BPO), designed to dynamically augment the knowledge depth of each sample. BPO is motivated by the observation that the usefulness of knowledge varies across samples, necessitating tailored learning of knowledge depth. To achieve this, we introduce gradient-based clustering, estimating the knowledge informativeness and usefulness of each augmented sample based on the model’s optimization direction. Our experimental results across various benchmarks demonstrate that BPO outperforms other baseline methods in alignment tuning while maintaining training efficiency. Furthermore, we conduct a detailed analysis of each component of BPO, providing guidelines for future research in preference data optimization.

## introduction reinforcement learning with human feedback (rlhf) @xcite has played a pivotal role in the success of large language models (llms) in recent years. it aims to align llms with human values and preferences during the post-training phase, leveraging extensive pairwise feedback from human annotators. to boost this alignment tuning, various advanced training techniques have been introduced, including direct preference optimization (dpo) @xcite , sequence likelihood calibration (slic) @xcite , and kahneman-tversky optimization (kto) @xcite . another line of effort aims to enhance the alignment process from a data perspective. this involves techniques such as data selection @xcite , sampling @xcite , and rearrangement @xcite to enhance various aspects of the original preference learning dataset, including quality @xcite , diversity @xcite , and difficulty @xcite . while existing data optimization methods focus on individual components of preference learning data (either the instruction or the response), a unified approach to systematically optimize preference data as a whole remains lacking. to address this problem, we begin by proposing the concepts of knowledge breadth and knowledge depth, which measure the comprehensiveness and depth of an llm or knowledge source respectively. building on these concepts, we conduct a systematic analysis of the resources allocated for enhancing the breadth and depth of knowledge during alignment tuning accordingly, uncovering a potential imbalance stemming from the structure of alignment datasets. to validate this imbalance, we propose a simple balance method, that includes knowledge breadth compression (kbc) and knowledge depth augmentation (kda), achieved by uniformly reducing the number of prompts and increasing the number of response pairs in the preference learning dataset. figure 1 demonstrates the overview process. the substantial improvement brought by both kbc and kda confirms the effectiveness of our balance approach in alignment tuning datasets. we further examine the preference learning dataset and observe the need to dynamically augment the knowledge depth of different samples. some samples in preference data contain rich, informative knowledge, while others consist of relatively shallow or less useful information. the former deserves more response pairs to allow llms to fully learn from them, whereas prior studies @xcite have shown that excessive focus on less useful knowledge can lead to negative outcomes. therefore, we propose balanced preference optimization (bpo), a method designed to dynamically augment the knowledge depth of each sample based on its informativeness. bpo first follows a general pipeline @xcite to sample a subset of representative and diverse prompts by clustering. next, we introduce gradient-based @xcite clustering to estimate the optimal knowledge depth for each sample from the perspective of model optimization. by clustering each augmented (prompt-response pair) based on their gradient features and selecting those closest to the cluster centers, bpo ensures that samples with gradient features near the centroids receive more focus. these centrally positioned samples are typically more informative and helpful for model optimization, making them deserving of greater knowledge-depth learning resources. extensive experiments across various benchmarks demonstrate both the effectiveness and computational efficiency of our bpo approach compared to other baseline methods. additional ablation studies and hyperparameter analyses confirm the effectiveness of each component, highlighting bpo's robustness across different hyperparameter settings. we also investigate alternative methods for dynamically estimating the required knowledge depth, with experimental results showing that the gradient-based approach performs best, achieving the highest human preference scores. to summarize, our contribution in this work is threefold: ## preliminary in this section, we first provide the definition of knowledge breadth and depth and bridge the connection between the two concepts with instruction and response numbers in alignment tuning datasets. then, we propose a simple strategy to balance the knowledge breadth and depth in the dpo tuning process and conduct preliminary experiments to evaluate its effectiveness. ## our method: balanced preference optimization in this section, we introduce our bpo method, which involves two core steps: knowledge breadth ## experiments in this work, we introduce our experiment settings and analyze our results. ## extra investigation & further analysis in this section, we explore how different clustering configurations affect bpo's performance, compare approaches for measuring knowledge depth, and analyze various response generation methods. ## conclusions in this work, we expose the potential imbalance between knowledge breadth and depth learning within preference learning datasets, and we conduct preliminary experiments using a simple balancing method to validate this observation. furthermore, we propose a novel balance preference optimization (bpo) method, which dynamically enhances the knowledge depth learning resources for each sample. bpo measures the knowledge depth of each sample from a unique model optimization perspective and dynamically selects varying numbers of response pairs for each preference sample. our experimental results on several alignment benchmarks demonstrate that bpo outperforms other preference optimization methods while maintaining high training efficiency. additionally, we provide an in-depth analysis of bpo, offering insights and guidelines on balancing preference learning datasets for future research.