A Systematic Examination of Preference Learning through the Lens of Instruction-Following (2025)

Abstract: In this work we systematically investigate how specific attributes of preference datasets affect the alignment and downstream performance of LLMs in instruction-following tasks. We use a novel synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with combinations of 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. With our synthetic prompts, we use rejection sampling (RS) and Monte Carlo Tree Search (MCTS) to obtain preference pairs. Then, we perform experiments investigating the effects of (1) the presence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the chosen, rejected responses and (3) the complexity of the training prompts. Our experiments reveal that shared prefixes provide marginal but consistent improvements and greater stability across challenging training configurations. While high-contrast preference pairs generally outperform low-contrast pairs, combining both often yields the best performance. Additionally, training on prompts of moderate difficulty leads to better generalization across different tasks. Our findings provide actionable insights into optimizing preference data curation for instruction-following tasks, offering a scalable and effective framework for enhancing LLM training and alignment.

## introduction aligning large language models (llms) with human preferences has remained a persistent challenge despite their recent success, particularly for tasks that involve generating nuanced, instructionfollowing responses. to address this bottleneck, preference learning has emerged as a vital technique applied in the final stages of llm posttraining @xcite @xcite . preference learning refines the * work done during internship at meta. ability of llms to align with human expectations by fine-tuning them on pairs of (chosen, rejected) responses. recent successes of using preference learning to develop frontier language models (ope-nai, 2023; @xcite @xcite have led to automated methods for curating preference pairs @xcite @xcite . while these techniques yield synthetic preference pairs that significantly improve model capabilities in closed-ended tasks, it is yet unclear which attributes of the preference pairs contribute to the improved alignment and downstream capabilities. existing research in preference learning has largely focused on the mechanics of optimization methods, such as direct preference optimization (dpo, @xcite ) and proximal policy optimization (ppo, @xcite ). while these methods are critical for training models, they operate with limited insight into how the structure, quality, and complexity of the preference datasets themselves affect outcomes @xcite . for example, questions remain about whether shared prefixes in paired responses improve learning, whether training on high-contrast pairs is always optimal, or how the difficulty of training prompts impacts generalization. without a systematic investigation of these factors, designing effective preference datasets is largely heuristic and suboptimal. in this work we seek to fill this gap by conducting a systematic investigation of how various attributes of automatically-curated preference datasets affect model performance. we approach this problem from the viewpoint of instructionfollowing @xcite @xcite , which are ideal for such analysis due to their complexity and capacity for integrating multiple constraints, giving us finer control over the data compared to other domains with binary correctness such as mathematics or tool we independently sample n different outputs from the policy, score each output with a verifier and take (high, low) scoring responses as the (chosen, rejected) pairs. mcts: we perform tree search with the policy while generating multiple actions. then, we use the rollouts from sibling nodes with (high, low) reward scores as the (chosen, rejected) pairs to obtain preference pairs with common prefixes up to the parent nodes. we first define 23 verifiable constraints spanning diverse requirements such as adherence to specific structural, stylistic, or formatting requirements, yet distinct from those presented in ifeval @xcite . these constraints form the foundation of our synthetic data generation pipeline, loosely inspired by instruct-skillmix @xcite , which (1) proposes new general-purpose prompts, (2) assigns valid combinations of verifiable constraints along with the associated parameters and (3) generates new prompts that incorporate mixtures of the constraints in natural language over a wide variety of domains. using this pipeline, we obtain a total of 48k unique prompts which contain mixtures of four, five or six verifiable constraints. using the synthetic prompts, we then apply two commonly-used methods for automatically curating preference pairs: rejection sampling (rs) and monte carlo tree search (mcts). rejection sampling presents a straightforward method to extract preference pairs for a given prompt by generating n independent responses with the policy model, scoring each response and using (high, low) scoring pairs of responses as the preference pairs @xcite . on the other hand, monte carlo tree search presents a more complex method for extracting preference pairs -for a given tree obtained via mcts, where each node repre-sents the partial response generated for the given prompt, any pair of sibling nodes with (high, low) scoring pairs of responses are used as preference pairs @xcite . the rs approach is computationally efficient but there is no structure to the generated responses. meanwhile, the mcts approach is more resource intensive, but returns pairs of responses that share common prefixes and a more nuanced contrast in their remaining suffixes. we use our synthetic prompts and the associated preference pairs to systematically investigate how different heuristics used for automatically curating preference pairs impact models' downstream performances. to this end, we focus on three critical dimensions that characterize a preference dataset: our findings reveal several actionable insights: unlike previous work, we conduct a systematic investigation into how different attributes of preference datasets affect the downstream performance. we choose instruction-following as our task to incorporate multiple constraints into the prompt and score the response on a fine-grained level. we use verifiable constraints to assign quality scores to our responses in a reliable and efficient manner. ## prompt synthesis we perform all our experiments with a new set of synthetic prompts that incorporate mixtures of verifiable constraints. these prompts allow us to evaluate the qualities of the generated responses in both a consistent and fine-grained manner, providing a suitable environment for us to control the attributes of the preference dataset. ## preference data curation using the prompts obtained in section 3, we generate responses and extract preference data using the correctness scores. we employ two methods for preference data curation: rejection sampling (rs) and monte carlo tree search (mcts). rs presents a straightforward and efficient way to obtain preference data but the (chosen, rejected) responses do not share any relationship. on the other hand, mcts is more expensive and slower to run, but returns (chosen, rejected) responses that share a common prefix with more nuanced contrast. rejection sampling. refer to figure 1 (left) for a visual overview. we first set a filtering criteria, where the chosen response must achieve a score of c according to our verifier and the rejected response must achieve a score of r. then, we generate n responses independently with the policy model and score each response with our verifier. given a prompt x, its associated set of verifiable constraints c, the response r and our verifier v which verifies whether the response satisfies any given constraint c, we compute the score as we extract all preference pairs such that 1) the chosen score equals c, 2) the rejected score equals r, and 3) there are no overlapping responses. monte carlo tree search. refer to figure 1 (right) for a visual overview. we conduct mcts at the token-sequence level -for a given prompt x and the mcts tree t , each node s i in t represents a partial response generated for x, and each edge (s i , s j ) in t , also known as an action, represents a sequence of tokens generated from s @xmath0. in this setup, we use an llm π as the policy and our verifier as the outcome reward model via three major steps: selection, expansion and backpropagation. selection. given the current node s t and k distinct actions (a 1 , ..., a k ) generated by the policy from s t , we balance exploitation and exploration to select the next node. each action is a sequence of tokens under a pre-specified maximum length. our selection depends on q(s t , a) and n (s t , a), the q-value and visit count of each subsequent node reached by taking action a from s t , respectively. we use the predictor+upper confidence bounds applied to trees (puct) and select the next node s t+1 according to the following formula: appendix c provides details on how to compute the policy score π(a i |s t ). using puct, we prioritize exploration during the initial stages of tree building and prioritize exploitation during the later stages of tree building as the visit counts increase. this results in a balanced trade-off between exploration and exploitation during our tree search. expansion. we perform expansion from the current node s t and generate k new actions with the policy π. for each new action, we perform m separate rollouts and score each rollout using a linear combination of the score v assigns and the self-evaluation score assigned by π. we use self-evaluation, denoted as π self-eval (s t ), in addition to the verifier scores, to provide step-level feedback during the tree search. again, refer to appendix c for more information on how to compute π self-eval (s t ). ## experiments and results we examine three dimensions of the preference dataset: the 1) existence of common prefixes between the (chosen, rejected) responses, 2) quality of the chosen and rejected responses, and 3) difficulty of the prompts used for training. we report results for additional experiments in appendix e. data curation setup. we implement rs by generating @xmath1.0, and score each output using v. meanwhile, we implement mcts with maximum depth of 5, number of actions 4, number of rollouts 4, c @xmath2.0 and λ = 0.2. training setup. we run our experiments with llama-3.1-8b-instruct (ai at meta, 2024) and finetune the model on the preference dataset using dpo @xcite . we finetune the model for one epoch with a maximum sequence length of 2048, learning rate of 5e-7 with a linear scheduler and 4 gradient accumulation steps. evaluation setup. we evaluate our models on ifeval @xcite , as well as three different synthetic evaluation sets designed to be more challenging than ifeval. our synthetic evaluation sets are created using the same pipeline in section 3.2, but with the ifeval constraints. we use @xmath3, 5, 6} for our evaluation sets and synthesize about 500 evaluation prompts for each value of k. using the finetuned models, we generate 16 responses to each prompt at temperature 0.7 and measure the average score. we use two metrics: 1) a hard score measuring whether the response satisfies all the constraints, and 2) a soft score measuring the ratio of the constraints satisfied by the response. ## conclusion we systematically investigate the effects of various attributes of preference datasets on model capabilities via instruction-following. to this end, we first build a data generation pipeline that combines general prompts with mixtures of verifiable constraints to synthesize challenging instructionfollowing prompts. we then automatically curate preference pairs using rejection sampling (rs) and monte carlo tree search (mcts). using the preference pairs, we examine the effects of (1) the existence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the responses, and (3) the complexity of the training prompts. our results indicate that having a common prefix in the preference pairs offers marginal yet consistent improvements, high-contrast preference pairs outperform low-contrast pairs but a mixture is sometimes better than both, and training on moderately difficult prompts is more helpful than training on extremely difficult prompts. our work provides a systematic framework for curating different types of preference datasets and sets the groundwork for future studies that extend the scope beyond verifiable instruction-following constraints to more general constraints.