Aligning Large Language Models by On-Policy Self-Judgment (2024)

Abstract: Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning. In this paper, we present a novel alignment framework, SELF-JUDGE that (1) does on-policy learning and 2) is parameter efficient, as it does not require an additional RM for evaluating the samples for on-policy learning. To this end, we propose Judge-augmented Supervised Fine-Tuning (JSFT) to train a single model to act as both a policy and a judge. Specifically, we view the pairwise judgment task, choosing the better response from a response pair, as a special case of the instruction-following task. The resulting model can judge preferences of on-the-fly responses from current policy initialized from itself. Experimental results show the efficacy of SELF-JUDGE, outperforming baselines in preference benchmarks. We also show that the rejecting sampling by itself can improve performance further without an additional evaluator.

## introduction research on aligning large language models (llms) with human preference has increasingly gained attention in recent years @xcite @xcite . reinforcement learning from human feedback (rlhf) is the most dominant approach for the alignment of llms for human preferences @xcite . it utilizes a reward model (rm) to estimate the human preference scores for the generated responses from policy. the policy is updated with on-policy reinforcement learning (rl) to maximize the estimated rewards of sampled responses regularized with the kl divergence between the current policy and reference policy. however, rlhf requires a complex setup for on-policy updates because of its simultaneous use of an rm with the reference model. in contrast, a line of research @xcite proposes discarding onpolicy learning and optimizing from preference orders in static datasets without rms (i.e., offline learning), whereas other studies propose constructing datasets with responses sampled from the initial policy and labeled by separated evaluators (i.e., offpolicy learning) @xcite @xcite @xcite . however, offline learning can lead to suboptimal results due to the lack of exploration @xcite , and off-policy learning could cause degeneration if an appropriate replay buffer strategy is not used @xcite . in this paper, we present self-judge, a simplified on-policy training scheme that trains a single model to perform on-the-fly feedback for current policy for improving itself. to this end, we introduce judge-augmented supervised fine-tuning (jsft) to obtain a model that can both generate a response and perform pairwise comparison, as shown in figure 1 . specifically, we regard the pairwise judgment task, choosing the better response from a response pair, as a special case of the instructionfollowing task, which can be answered in a single token and optionally with a rationale. self-judge initializes the current policy and reference policy from jsft's resulting model. self-judge samples a response pair from the current policy and chooses a better response in the pair by the reference policy, then updates the current policy with preference orders without an rm @xcite @xcite . experimental results show that self-judge outperforms rlhf and other offline and off-policy approaches @xcite @xcite on preference benchmarks. unlike existing approaches, self-judge leverages on-policy learning while not introducing an additional evaluator. the results demonstrate the effectiveness and parameter efficiency of self-judge performing on-policy self-training for llm alignment. furthermore, we show that self-judge can maximize performance through self-rejection that selects the best response from its own responses using its judgment capabil-ities learned through jsft. in particular, the performance gains are significant when the pairwise judgment task of jsft involves comparisons based on principles with rationale for the decision. in summary, our main contributions are: ## preliminaries aligning by reinforcement learning to align llms with human preferences, rlhf (ziegler et al., 2020) follows three stages: 1) obtaining an initial policy π ref through sft from the pre-trained llm, 2) training an rm from human preference triplets (x, y w , y l ), where y w is a chosen response, and y l is a rejected response for a given prompt x, 3) fine-tuning the initial policy by proximal policy optimization (ppo) @xcite , with kl divergence between current policy π and reference policy π ref as a regularization for the reward maximization objective. generally, rms are trained by bradley-terry model (bradley and terry, 1952), which minimizes negative log-likelihood of score difference, logσ(r θ (x, y w )r θ (x, y l )), to compute a pointwise scalar reward for the response sampled from the current policy. this on-policy rollout procedure provides the frequent exploration of responses but introduces an additional training stage and memory usage due to the rms. aligning from preference orders it has been observed that rm is susceptible to language biases such as the response length, preferring longer responses over shorter ones @xcite . @xcite suggest a language modeling loss on preferred context (x, y w ) during the training of the rm to address this issue. this approach, however, necessitates another pre-training stage of the rm. direct preference optimization (dpo) @xcite and sequence likelihood calibration with human feedback (slic-hf) @xcite introduce objectives that can be trained by only preference orders y w ≻ y l without scalar rewards and need for rms. however, offline learning methods optimized on static datasets inherently reach sub-optimal results compared to online learning @xcite . @xcite ; @xcite propose an online learning approach that iteratively constructs datasets by responses sampled from the policy. however, this off-policy approach with a large buffer size can induce performance degeneration @xcite . ## self-judge in this section, we describe our proposed alignment framework, self-judge, which utilizes a single model that acts as both policy and judge: sampling responses and judgment over response pairs. it can provide feedback on the response pairs sampled from the current policy for improving itself in an on-policy manner and also perform rejection sampling by itself, as depicted in figure 2 . ## discussion concurrent works recent works such as self-rewarding language models @xcite and oaif @xcite have intersections with self-judge in aspects of self-training and on-policy learning for pairwise preferences. self-rewarding language models trains the initial policy to act as a judge, but evaluations are performed through chain-of-thought reasoning, which requires a significant computational cost for evaluation. therefore, off-policy learning with an iterative training scheme was adopted, which can yield inferior results compared to on-policy learning, as shown in ## conclusion we propose a parameter-efficient on-policy preference alignment framework, self-judge, introducing judge-augmented supervised fine-tuning (jsft). a model trained by jsft can perform feedback to the current policy for improving itself by acting as a judge. this self-training does not require additional training stages and parameters for a reward model during the policy updates. our resulting model outperforms rlhf, offline, and offpolicy baselines in preference benchmarks, demonstrating the advantages of on-policy learning and parameter efficiency of self-judge. besides, we provide various analyses on the best configurations and efficacy of the proposed jsft. specifically, jsft boosts performance as a judge, and involving comparisons based on principle with rationale about decision leads to further improvement. this enhanced judging capability leads to further selfimprovement as a policy at inference time by selfrejection over the model's own responses.