Teaching Language Models to Self-Improve by Learning from Language Feedback (2024)

Abstract: Aligning Large Language Models (LLMs) with human intentions and values is crucial yet challenging. Current methods primarily rely on human preferences, which are costly and insufficient in capturing nuanced feedback expressed in natural language. In this paper, we present Self-Refinement Tuning (SRT), a method that leverages model feedback for alignment, thereby reducing reliance on human annotations. SRT uses a base language model (e.g., Tulu2) to generate initial responses, which are critiqued and refined by a more advanced model (e.g., GPT-4-Turbo). This process enables the base model to self-evaluate and improve its outputs, facilitating continuous learning. SRT further optimizes the model by learning from its self-generated feedback and refinements, creating a feedback loop that promotes model improvement. Our empirical evaluations demonstrate that SRT significantly outperforms strong baselines across diverse tasks and model sizes. When applied to a 70B parameter model, SRT increases the win rate from 9.6% to 25.8% on the AlpacaEval 2.0 benchmark, surpassing well-established systems such as GPT-4-0314, Claude 2, and Gemini. Our analysis highlights the crucial role of language feedback in the success of SRT, suggesting potential for further exploration in this direction.

## introduction recent advances in large language models (llms) have revolutionized the field of natural language processing. these models have demonstrated remarkable capabilities in various tasks such as open-ended generation, question answering, and mathematical reasoning @xcite @xcite . however, despite their impressive performance, llms occasionally generate content that can be untruthful or harmful. this highlights the need to align language models with human intentions and values to ensure safe and controllable deployment @xcite @xcite . many current methods to align llms, such as reinforcement learning from human feedback (rlhf) and direct preference optimization (dpo), rely on human preferences @xcite @xcite . these preferences are typically expressed through rankings or scores. however, there are two main challenges with these methods. firstly, annotating human preferences is expensive and time-consuming, which makes it difficult to scale these techniques. for instance, @xcite spent $40k to annotate approximately 60k feedback instances for a single summarization task. secondly, simple rankings or scores may not fully capture the subtlety and complexity of human preferences. this can limit the depth of feedback provided to models for improvement @xcite . humans, on the other hand, can derive insights from minimal language feedback, indicating that there is potential for more sophisticated and efficient alignment methods @xcite . in response to these challenges, we introduce a new method for aligning language models, which we call self-refinement tuning (srt). srt obviates the need for human preferences with a twostage learning process. in the first stage, srt employs a powerful model like gpt-4 to critique and refine the outputs from a base model, such as tulu2 @xcite . the base model is then finetuned on critiques and refinements, enabling selfevaluation and improvement. in the second stage, srt further boosts the model by learning from selffeedback. specifically, srt uses the fine-tuned model to generate model preferences, i.e., pairs of outputs and refinements. subsequently, srt optimizes the model on these preferences using dpo @xcite . we evaluate srt on open-ended generation, question answering, and mathematical reasoning. empirical results show that srt consistently outperforms baseline models of sizes from 7b to 70b, with an average performance enhancement of 3.7 to 4.0 points. these improvements are obtained using merely 22k feedback instances annotated by gpt-4 turbo. figure 1 shows that the srt significantly improves tulu2 models using modelgenerated feedback. the strongest model trained with srt attains a 25.8% win rate against gpt-4 turbo on the alpacaeval 2.0 benchmark. this performance surpasses established systems such as gpt-4 0314, mistral medium, claude, and gemini. our analysis confirms that the success of srt primarily stems from its language feedback feature, which identifies weak areas and offers valuable suggestions for improvement. ## related work this section briefly reviews two critical areas in the alignment of llms: learning from ai feedback and learning to self-improve. our work intersects with these domains and addresses existing challenges. learning from ai feedback. learning from human feedback is the key to the success of stateof-the-art language models such as gpt-4 (ope-nai, 2023) and gemini @xcite . however, acquiring high-quality human feedback is both costly and time-consuming @xcite @xcite . this has led to a growing interest in harnessing ai-generated feedback to enhance llms @xcite @xcite . for instance, @xcite employs llms to annotate ranking-based preferences. our research diverges from this approach by utilizing a more comprehensive range of feedback, encompassing identified weaknesses and proposed improvements. a similar approach is reinforcement learning from ai feedback (rlaif, @xcite , which uses llms to generate critiques and refinements. however, rlaif encounters challenges such as computational inefficiency and unstable training dynamics due to the inherent complexities of reinforcement learning techniques @xcite . we address these challenges by unifying the generation of feedback and refinement into instruction-following, thereby simplifying the training process. learning to self-improve. using large language models (llms) to identify and correct their own errors has become increasingly popular. this self-improvement capability has significantly enhanced performance across various tasks, such as question-answering, code generation, and mathematical reasoning @xcite @xcite . @xcite conducted a comprehensive survey of this field. however, these approaches rely on critique and refinement skills that are generally lacking in opensource language models @xcite . this underscores the urgent need to train open-source models for selfimprovement. unlike previous methods that train separate models for generation, critique, and refinement @xcite , our approach employs a single model for all tasks, facilitating knowledge transfer across them. additionally, our method is designed for the general alignment of llms, in contrast to prior methods that are task-specific @xcite @xcite . ## methodology figure 2 presents the two-stage self-refinement tuning (srt) process. srt aims to enhance the capabilities of a base language model (denoted as m base ) by harnessing learning from ai feedback (laif), thereby reducing the reliance on human feedback. in the first stage, we train m base to selfimprove by learning from feedback and refinements annotated by more powerful models. in the second stage, we further optimize the model using its selfgenerated feedback. ## analysis this section provides an in-depth examination of the factors that affect the performance of our methods. initially, we evaluate the efficacy of self-refinement in srt by contrasting it with the widely adopted re-ranking approach. subsequently, we explore the influence of various elements in srt, encompassing language feedback, the quality of refinements, and the quantity of training data. through these analyses, we aim to deepen the understanding of our methods. ## conclusion we have presented self-refinement tuning (srt) for aligning language models using language feedback. srt initially teaches language models to assess and enhance their outputs by learning from feedback provided by more advanced models. subsequently, srt optimizes these models further by learning from their self-generated feedback. the primary benefits of srt are twofold: (1) it obviates the need for human-annotated preferences, and (2) it obtains promising performance across a wide range of tasks and model sizes. in our analysis, we find that both high-quality language feedback and refinements are crucial for language models to learn to self-improve. collectively, we demonstrate that srt is an effective and efficient strategy for improving the performance of language models.