Reinforcement Learning with Supervised Alignment (2025)

Abstract: Supervised fine-tuning (SFT) is a widely used and highly effective method for adapting Large Language Models (LLMs) to specific tasks. However, it often suffers from overfitting, causing models to excel on fine-tuned data but struggle with unseen or rare real-world inputs. While recent methods like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with AI Feedback (RLAIF) aim to align LLMs with human values and tasks, they face challenges such as the high cost of human labeling or instabilities and biases inherent in using LLMs as judges. To address these issues, we propose a novel approach called Reinforcement Learning from supervised Alignment (RLA), which constructs a supervised alignment to train the reward model for reinforcement learning. Using only 100,000 MS MARCO samples, our method outperforms RLAIF by a relative margin ranging from +5.38% to +131.8%. It also significantly enhances the baseline Llama3 LLM, achieving up to +55% improvement on in-domain tasks and up to +16% on out-of-domain tasks. While RLA slightly underperforms supervised fine-tuning (SFT) on in-domain benchmarks, it surpasses SFT by up to 50 times on out-of-domain and cross-task evaluations, demonstrating strong generalization capabilities.

## introduction supervised fine-tuning (sft) is a crucial technique for adapting large language models (llms) to specific tasks to enhance their accuracy, which is widely applied to many downstream applications such as text processing and analysis. however, sft is prone to overfitting when the model adapts too closely to the fine-tuning dataset, losing its ability to generalize. this results in a model that performs well on familiar data but struggles with new, unseen, or more varied real-world scenarios. to address these challenges, there have been methods proposed to enhance sft generalization @xcite @xcite . however, these methods either require capable base models that can follow user instructions, or may decrease their effectiveness under more complex tasks or restricted time constraints. reinforcement learning with human alignment in instructgpt @xcite combined with rag @xcite have significantly improved model responses' factuality and alignment with user intentions. however, human alignment faces three main challenges: the timeconsuming and costly nature of labeling human instructions, the complexity and potential bias in human values and intentions, and the inconsistency in individual labelers' interpretations. these issues have led researchers to explore ai feedback (rlaif) approaches, such as rain @xcite , second thoughts @xcite , and selfrewarding @xcite , which use llms for reward modeling. while these methods address some human alignment limitations, they introduce new concerns, including instability and error propagation due to the lack of verified truth integration in the learning process. our study presents a novel reinforcement learning with supervised alignment (rla) that automates the alignment process, eliminating the need for human intervention or llm-as-judge. rla utilizes an existing dataset as its foundation. for each sample within the dataset, our target llm (to be improved on) is queried to generate a set of responses. subsequently, the semantic similarity between each response and the sample's label is measured. the underlying hypothesis posits that the semantic similarity value correlates positively with response quality, such that higher similarity indicates superior responses. following this assessment, the responses are subjected to pairwise ranking based on their similarity scores. this ranking process facilitates the gen-eration of training data, which is then employed to train a reward model. this automatically derived reward model serves as a substitute for the conventional human-aligned reward typically used in rlhf. this innovative approach potentially enhances the efficiency and scalability of the llm alignment process, while keeping its verifiability. formally, our method is composed of the following three procedures as illustrated in figure 1: 1. supervised alignment ∆ first, we construct a preference-based dataset, we call supervised alignment, based on an existing linguistic dataset @xmath0, y true }, with pairs of queries @xmath1. we input queries to our target llm π ϕ ref and generate k distinct responses. then, we compute the cosine similarity c sim (y ik , y ′ ik ) between the embeddings of each response and its label, using this similarity as a proxy for preference to organize the k responses into k 2 "chosen" and "rejected" pairs. each response pair with its query is added to the supervised alignment dataset (∆). the procedure is described in algorithm 1. ## related work our method inherits the rlhf framework and supervised learning's data observation and aim to improve them in the following perspectives: sft: both sft @xcite @xcite @xcite and our method aim to maximize the likelihood of sample labeling in existing datasets. however, our method achieves better generalization than sft by leveraging the rlhf framework, where the generalized advantage estimation (gae) @xcite algorithm and trusted region policies provide more stable and generalized results compared to applying gradient ascent directly on the training data, by addressing temporal dependencies and credit assignment, and reducing variance and overfitting in gradient estimates, leading to noise resilience and low variance. additionally, the clipping mechanism in proximal policy optimization (ppo) @xcite prevents drastic shifts in the learned distribution, reducing the risk of converging to local optima. lastly, the ranking strategy in reward dataset construction mitigates numerical instability by focusing on relative differences among candidates, making the learning process more robust. ## methods our method automatically generates the alignment by computing the similarity of our query responses and its labels in the existing dataset. in following, we will describe its three main components: the supervised alignment ∆ , the reward model (σ ∆ ), and the reinforcement learning framework rla. ## conclusion we introduce rla that trains the reward model used in reinforcement learning leveraging a existing labeled dataset. our approach improves the accuracy of the qa across data sets in the domain and out of the domain and in cross-tasks few-shot and zero-shot scenarios, resulting in a more generalizable learning method.