Reasons to Reject? Aligning Language Models with Judgments (2024)

Abstract: As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval using LLaMA2-13b. CUT can also align LLMs in an iterative fashion using up-to-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval using LLaMA2-chat-13b. Further analysis suggests that judgments hold greater potential in LLM alignment than rewards.

## introduction large language models (llms) acquire substantial world knowledge and reasoning capabilities through large-scale pre-training @xcite @xcite . to unleash the power of pre-trained llms for realworld applications, it is crucial to ensure that llms can follow human preferences and values @xcite . this process, known as alignment, is critical for making artificial intelligence a helpful and reliable ally for humanity @xcite . figure 1 illustrates three paradigms to achieve alignment. the most straightforward one is learning from demonstrations, wherein demonstrations of desired responses to a set of instructions are collected and used to fine-tune llms in a supervised fashion @xcite . however, the performance gains diminish rapidly as the data size increases @xcite . in contrast, learning from feedback (rewards or judgements) offers a more scalable approach @xcite . one significant advantage of feedback over demonstrations is that feedback can convey both positive and negative aspects, enabling the model to discern desirable and undesirable outcomes. in addition, feedback is tailored to the current model, adhering to the principle of teaching according to the learner's aptitude. prior research on learning from feedback primarily focuses on value feedback (i.e., scalar rewards), employing reinforcement learning (rl) algorithms, such as ppo @xcite , to optimize an llm to maximize the rewards of its generated responses. nevertheless, scalar rewards are information-sparse for solely indicating the goodness of a response. on the other hand, language feedback (i.e., judgment) can offer more nuanced commendations and critiques through natural language expressions @xcite . specifically, judgments can elucidate the specific aspects that are good or bad, the rationale behind their evaluation, and suggestions for improvement. the above suggests that aligning llms with judgments can be more advantageous. in this study, we present an extensive investigation of potential methods that can be adapted for aligning llms with judgments. to facilitate a comprehensive aligning process, we propose a novel framework, contrastive unlikelihood training (cut), that enables fine-grained inappropriate content detection and correction based on judgments. cut detects inappropriate content in a response by contrasting its generation probabilities under aligned and misaligned conditions and further penalizes the inappropriate content with unlikelihood training @xcite . we carry out experiments for both offline and online alignment, wherein the target llm learns from the off-the-shelf judgments and the judgments derived from self-generated responses, respectively. extensive results on offline alignment demonstrate the effectiveness of cut in learning from judgments in both cold-start (using unaligned base llms such as llama2) and warm-start (using aligned base llms such as llama2-chat) scenarios. notably, when trained with only 1317 offline judgment data, cut attains a winning rate of 61.06 and outperforms the best baseline by 50.84 points on alpacaeval using llama2-13b. furthermore, our online alignment experiments show that cut is capable of iteratively refining llms using model-specific judgments, with a steady performance improvement from 81.09 to 91.68 points on alpacaeval using llama2-chat-13b. our analysis comparing rewards and judgments suggests that aligning llms with judgments offers significant potential and warrants future research. our contributions can be summarized as follows: 1) we present the first systematic exploration of aligning llms with judgments. 2) we introduce a novel framework, cut, that facilitates the alignment of llms through fine-grained inappropriate content detection and correction based on judgments. 3) our results showcase the effectiveness of cut in aligning llms across cold-start and warm-start scenarios, generalist and specialist applications, as well as offline and online settings. 4) our analysis indicates that judgments hold greater potential over rewards for aligning llms. ## related work existing approaches for learning from feedback can be classified into two distinct categories: prompting and fine-tuning, differentiated by whether updates to the llms' parameters are absent or present. prompting. prompting does not alter the parameters of llms. instead, it leverages judgments on previous responses to elicit improved responses from llms @xcite . judgments can be sourced from diverse aspects @xcite and the refinement process can be iterated multiple times @xcite @xcite . however, these methods rely on the in-context learning capabilities of the llms and consume more computation than single-pass generation @xcite . fine-tuning. fine-tuning aims to train an llm that can generate better responses immediately. scalar rewards have been extensively used through the lens of rl, particularly ppo @xcite @xcite . however, ppo is known to be complex and unstable @xcite , which has attracted numer-ous efforts to simplify or stabilize the training process @xcite @xcite @xcite @xcite . another strand of work, named hindsight @xcite , transforms scalar rewards to language instructions and teach llms to generate responses of different qualities. there are also attempts to leverage the results of prompting for training a better model. that is, the improved response elicited by the judgment is employed as new training data @xcite @xcite . however, these methods still suffer from the incapability to learn from mistakes, which is the core spirit of learning from feedback. ## preliminaries in this section, we first lay out a formal problem definition of aligning llms with judgments and then present a survey of three potential methods that can be adapted for tackling this problem. ## contrastive unlikelihood training to overcome the limitations mentioned in § 3, we propose cut, a fine-tuning framework to align llms with judgments. the core idea of cut is summarized as learning from contrasting. we contrast the response generation under different conditions to shed light on the appropriate behavior that the llm should keep, as well as the specific content necessitating adjustments. based on these insights, we use mle for appropriate content and ut @xcite for inappropriate content. ## experiments eq. 4 @xcite ; (3) ilf-mle described in eq. 3 @xcite , and ilf-dpo @xcite that change the learning objective from mle to dpo. the details of the model implementations are provided in appendix a.1. ## conclusion we systematically explored the alignment of llms through the lens of judgments. we investigated three potential methods that can be adapted for aligning llms with judgments but found them unable to fully capitalize on judgments. we proposed a novel framework cut, that enables direct and explicit learning from judgments and facilitates fine-grained inappropriate content detection and correction. extensive evaluations demonstrated the effectiveness of our cut in various settings, including offline and online, specialist and generalist, as well as cold-start and warm-start scenarios. for example, the online alignment experiments showed that cut can iteratively improve llms with upto-date model-specific judgments, akin to how humans progressively refine their behaviors through ongoing feedback. our analysis comparing rewards and judgments suggested that aligning llms with judgments is a promising research area.