Self-Evolution Fine-Tuning for Policy Optimization (2024)

Abstract: The alignment of large language models (LLMs) is crucial not only for unlocking their potential in specific tasks but also for ensuring that responses meet human expectations and adhere to safety and ethical principles. To address the challenges of current alignment methodologies, we introduce self-evolution fine-tuning (SEFT) for LLM alignment, aiming to eliminate the need for annotated samples while retaining the stability and efficiency of SFT. SEFT first trains an adaptive reviser to elevate low-quality responses while maintaining high-quality ones. The reviser then gradually guides the policy’s optimization by fine-tuning it with enhanced responses. The method excels in utilizing unlimited unannotated data to optimize policies via supervised fine-tuning. Our experiments on AlpacaEval and MT-Bench demonstrate the effectiveness of SEFT and its advantages over existing alignment techniques.

## introduction recent years have showcased the remarkable capabilities and performance of large language models (llms) across a broad range of tasks. these capabilities are attributed not only to their vast parameter sizes and the extensive text corpora used for pre-training @xcite but also to the critical process of aligning these models with human expectations @xcite . such alignment is essential to ensure that the outputs of llms are helpful, honest, and harmless @xcite across various tasks and applications. the pursuit of aligning llms with human preferences has led to three main methodologies: supervised fine-tuning (sft), reinforcement learning from human feedback (rlhf) @xcite , and offline rlhf. sft fine-tunes llms on downstream tasks using instruction-following data to guide them in producing responses that * corresponding author. match the dataset's ground truth @xcite . rlhf employs a sophisticated approach by first training a reward model that assigns higher rewards to responses aligning better with human preferences, and then optimizing the llm policy using policy-gradient methods such as proximal policy optimization (ppo) @xcite . in offline rlhf, exemplified by direct preference optimization (dpo) @xcite , the policy is directly optimized using pre-collected preference data, omitting the need for a reward model. this aims to maximize the probability of producing chosen responses and minimize rejected ones. each of these methods comes with its strengths and weaknesses. sft, while efficient, is hindered by the scarcity of high-quality human-annotated data and tends to suffer from poor adaptability to out-of-distribution samples @xcite . rlhf demands substantial computational overhead for training an additional reward model @xcite and faces optimization challenges such as inefficiency and instability. of-fline rlhf methods, which are directly optimized on preference data without the need for a reward model, tend to suffer from distribution drift issues and may lead to biased policies that favor out-ofdistribution responses @xcite . in response to these challenges, this paper introduces a novel self-evolution fine-tuning (seft) method for policy optimization. seft first trains a preliminary reviser on preference data, which takes a prompt and the raw response of an llm as input and aims to output a higher-quality response. however, our initial experiments suggest that the preliminary reviser tends to revise the original responses indiscriminately, regardless of its capabilities, which may degrade the quality of high-quality responses. as illustrated in figure 1 , the reviser generally enhances low-quality responses but occasionally degrades high-quality ones. therefore, we continue to train an adaptive reviser that learns to assign a revision label based on the difficulty of revising the initial response: [major revise] indicates a substantial revision, [minor revise] signifies a minor revision, and [no revise] means that no revision is needed. the revision labels guide the reviser to make feasible revisions and refrain from attempting those beyond its capability, ensuring the overall quality of revised responses. the adaptive reviser then assesses the policy's outputs and improves low-quality responses to high-quality ones, which supports the subsequent fine-tuning of the policy. this adaptive mechanism aligns the policy with human preferences without the need for exhaustive annotated data or complex explorations. the rationale behind seft is that it utilizes the pseudolabels generated by powerful llms for fine-tuning. studies @xcite have demonstrated effective fine-tuning of models with high-quality synthetic labels from robust llms like gpt-4, aligning the policy with human-like responses affordably through approximate human annotations. to evaluate the proposed seft, we implemented the adaptive reviser on nectar @xcite using renowned llms of various scales. we then performed policy optimization with zephyr-7b-sft-full @xcite as the base model on ultrafeedback @xcite , where only the prompts were used. the optimized policy was evaluated on the alpacaeval 2.0 @xcite and mt-bench @xcite benchmarks. experimental results demonstrate the effectiveness of seft and highlight its superiority over traditional alignment methods such as sft, dpo @xcite , and orpo @xcite . the results also reveal that the adaptive reviser can effectively assess the difficulty of revising responses and enhance the overall quality of model outputs. furthermore, experiments with additional unlabeled data show that incorporating more unlabeled data consistently enhances the performance of seft. ## related work this section reviews mainstream llm alignment methods and shows how our approach diverges. ## seft: self-evolution fine-tuning self-evolution fine-tuning (seft) aims to provide a robust and efficient solution for policy optimization. as shown in figure 2 , we first train an adaptive reviser that evaluates the initial responses generated by the policy and makes adaptive revisions to enhance the overall quality. then, the policy is fine-tuned using these enhanced responses. ## experiments we conduct extensive experiments to evaluate seft's effectiveness in enhancing initial response quality and optimizing the policy, while also analyzing the benefits of integrating unlabeled data and the impacts of progressive policy optimization. ## conclusion in this paper, we introduce self-evolution finetuning (seft) for alignment of large language models (llms). seft employs an adaptive reviser to enhance the overall quality of initial responses by making revisions only when improvements are feasible. our experiments show that the adaptive reviser consistently enhances response quality and outperforms baseline methods. the effectiveness of seft in policy optimization is validated through extensive experiments on benchmarks such as al-pacaeval 2.0 and mt-bench. notably, we highlight seft's capability to leverage unannotated data to improve response quality while maintaining stability and efficiency in policy optimization.