Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models (2025)

Abstract: The key to effective alignment lies in high-quality preference data. Recent research has focused on automated alignment, which involves developing alignment systems with minimal human intervention. However, prior research has predominantly focused on developing data generation methods, while insufficient attention has been paid to quality control mechanisms and often produces inaccurate and unhelpful data, leading to unpredictable benefits during iterative optimization. In this paper, we present Self-Steering Optimization ( SSO ), an algorithm that autonomously generates high-quality preference data, eliminating manual annotation requirements. SSO employs a specialized optimization objective to build a data generator from the policy model itself, which is used to produce accurate and on-policy data. We demonstrate SSO ‘s effectiveness through comprehensive experiments on two series of models: Llama 3 and Qwen 2. Our evaluation across diverse benchmarks shows that SSO consistently outperforms baselines in human preference alignment and reward optimization. Further analysis validates SSO as a scalable framework for preference optimization, benefiting the advancement in automated alignment techniques.

## introduction the field of natural language processing has undergone revolutionary advancements driven by large language models (llms). after meticulous alignment processes, llms have demonstrated remarkable capabilities in following instructions and understanding human preferences. this leads to the development of widely acclaimed products like chatgpt @xcite , which have captured significant public attention. however, aligning llms with human preferences is not trivial. despite the existence of proximal policy optimization * corresponding authors. consequently, recent researchers have shifted their focus towards automated alignment, with the intention of developing scalable, high-quality alignment systems with minimal human intervention. the cornerstone of this paradigm is the pursuit of scalable alignment signals that are capable of effectively replacing human-annotated preference data. current popular strategies include selfjudgement @xcite , principle-based automated alignment @xcite , constitutional ai @xcite , and other methods @xcite . however, these methods do not pay enough attention to quality control mechanisms. the ideal preference data, as defined in rlhf @xcite , is constructed through human annotation, where the responses are sampled from the policy model and ranked according to their qualities. as shown in figure 1 (a), this type of preference data demonstrates high accuracy foot_0 and on-policy nature foot_1 , leading to ideal alignment optimization (tajwar et al. @xcite . however, automated methods often fail to simultaneously guarantee accurate preferences and on-policy responses, instead producing suboptimal data, as shown in figures 1(b ) and 1(c), which impedes model alignment. for example, self-judgment is hampered by the inherent limitations of the model; this judging ability is restricted and difficult to improve, often resulting in hacked rewards and inaccurate preference data in figure 1 (c) @xcite . in other methods, incorporating additional input or processes may lead to off-policy responses and sub-optimal preference data, as shown in figure 1(b) . we then recognized the need for a novel approach to generate high-quality preference data to address these limitations and advance automated alignment. one problem is to control the distribution of the chosen and rejected responses when constructing preference data. for most automated alignment methods, this seems an impossible task because chosen and rejected responses are typically obtained through complex pipelines. however, we found that principle-based methods @xcite can achieve this goal because they construct preference data by directly sampling from the policy model based on good and bad principles. it can be approximated that the dis-tributions of the chosen and rejected responses π w and π l are the distribution of the policy model π θ with good and bad principles. in this work, we introduce self-steering optimization (sso), a pioneering method that automatically generates accurate and near-on-policy preference data for the policy model. sso optimizes a data generator with a special loss to control the distributions of chosen and rejected responses, and then uses the data produced by this generator to further optimize the policy model. specifically, sso first prompts the policy model with original queries x and a set of contrastive principles p + and p -for the responses as training data and then optimizes the policy model based on two key objectives: a) making rejected responses approximately on-policy to ensure the effectiveness of negative gradients in subsequent optimization. we only take care of the rejected responses, as the chosen responses usually have high generation probabilities. b) maintaining a consistent gap between the chosen and rejected responses to ensure the accuracy of the preference data. we demonstrate the effectiveness of self-steering optimization on qwen2 @xcite ) and llama3 (llama team, 2024) backbones. our experiments reveal sso's ability to generate accurate and on-policy preference data. as a result, improvements are observed on a wide range of benchmarks such as math @xcite , ifeval @xcite , mt-bench @xcite , and alpacaeval 2.0 @xcite . furthermore, we conducted experiments through reward optimization, which also achieved satisfying results. without human annotation or external models, sso even outperforms baselines with annotated data @xcite , underscoring its potential as a scalable and efficient alignment approach. automated alignment previous alignment studies rely on manually annotated preference data and algorithms such as rlhf and dpo to conduct model alignment. recently, numerous studies have found that llm-generated data can reach the quality of ordinary manual annotations @xcite . these findings increased the attention of automated alignment @xcite . automated alignment aims to minimize human intervention by addressing the prohibitively expensive cost of human annotation. current methods can be divided into four types based on the source of alignment signals @xcite : 1) inductive bias, from introducing appropriate assumptions and constraints @xcite @xcite @xcite . 2) behavioral imitation, another aligned model @xcite @xcite . 3) model feedback, feedbacks from other models @xcite . 4) environmental feedback, environmental interaction @xcite . ## conclusion in this work, we proposed a novel approach called sso (self-steering optimization) to enhance model alignment by generating accurate and on-policy preference data without additional human annotations. sso applying two specific losses l + and l -to control the distribution of the chosen and rejected responses, respectively, to ensure the effectiveness of negative optimization and maintain the precision of preference data. we conducted extensive experiments on the qwen2 and llama3 backbones to evaluate the effectiveness of sso in model alignment, demonstrating significant improvements on various subjective and objective benchmarks, including alpacaeval 2.0, mt-bench, ifeval, etc. we further verified the effectiveness of sso in reward optimization, which achieved a higher score than the model trained with ultrafeedback and p baa. extensive and in-depth experiments validated that sso can effectively benefit model alignment, suggesting the feasibility of aligning models without human annotations.