Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation (2024)

Abstract: Aligning large language models (LLMs) with human expectations without human-annotated preference data is an important problem. In this paper, we propose a method to evaluate the response preference by using the output probabilities of response pairs under contrastive prompt pairs, which could achieve better performance on LLaMA2-7B and LLaMA2-13B compared to RLAIF. Based on this, we propose an automatic alignment method, Direct Large Model Alignment (DLMA). First, we use contrastive prompt pairs to automatically generate preference data. Then, we continue to evaluate the generated preference data using contrastive prompt pairs and calculate a self-rewarding score. Finally, we use the DPO algorithm to effectively align LLMs by combining this self-rewarding score. In the experimental stage, our DLMA method could surpass the RLHF method without relying on human-annotated preference data.

## introduction with the significant enhancement in the capabilities of llms, various models represented by chatgpt have demonstrated outstanding abilities in multiple fields, including machine translation @xcite , code generation @xcite , and dialogue systems @xcite . however, a key challenge is ensuring that the outputs of these llms align with human expectations, thereby producing more helpful and harmless results. this requires the llms to provide not only accurate information but also consider attributes such as helpfulness and harmlessness. recent studies have shown that reinforcement learning from human feedback (rlhf) @xcite in contrast to the rlaif method @xcite for evaluating preference data with textgeneration based method, our approach evaluates preference data through the comparison of output probabilities between responses under two contrastive prompts. @xcite is an effective approach for aligning llms with human expectations, which involves training a reward model with human-annotated preferences datasets, followed by reinforcement learning training of the llm using this reward model. however, human annotation is highcost and its implementation in complex scenarios presents a challenge. for instance, this challenge is evident in the "super alignment" scenario proposed by openai @xcite . therefore, aligning llms with minimal human supervision has emerged as a critical issue. to address this challenge, many efforts have been devoted to enabling llms to construct pref-erence datasets by themselves. for instance, the rlaif @xcite prompts llms to generate multiple responses to a given question and then evaluates these responses based on pre-defined rules. however, the variability in responses generated from a single question may be insufficient. context distillation (cd) @xcite incorporates additional system prompts, guiding the model towards better responses, which are then used to train the llm through supervised learning to distill the effectiveness of the system prompts. nevertheless, the supervised learning approach may influence the model's capabilities. consequently, the rlcd @xcite introduces the generation of preference data through contrastive system prompts (e.g., one prompting less harmful output and another more harmful), followed by training a reward model to align the llm with reinforcement learning. however, due to randomness during the llm's text generation, the produced preference data could be noisy. therefore, the self-generated preference data still necessitates evaluation. the rlaif @xcite proposes designing prompts to let llm judge which response is better (as illustrated in figure 1 ). however, it places high demands on the capabilities of llms and cannot ensure accuracy. in this work, we discover that the quality of response pairs produced by llms could be assessed by examining the output probabilities for each response under corresponding contrastive prompts (probability-based evaluation). moreover, we prove that the probability-based evaluation is more accurate than the text-generation based evaluation in experiments based on llama2-7b and llama2-13b @xcite with llmgenerated text. based on these findings, we propose our direct large model alignment (dlma) method through self-rewarding contrastive prompt distillation. our method is divided into three steps. initially, we employ a contrastive prompt pair to let the llm generate a pair of responses for a question. subsequently, we assess the quality of the response pair by comparing their output probabilities under contrastive prompts and calculating a selfrewarding score. finally, we utilize an revised direct preference optimization (dpo) @xcite algorithm, incorporating this self-rewarding score, to effectively align the llm. our experiments validate the effectiveness of dlma. based on llama2-7b and llama2-13b models, dlma surpasses existing baselines on pku-saferlhf @xcite , hh-harmless, and hh-helpful @xcite benchmarks without requiring manually annotated preference data. analysis confirms the self-rewarding score's accuracy in reflecting preference relationships. remarkably, dlma achieves effects better than rlhf aligned results with human annotated preference data. furthermore, we verify that the alignment process does not degrade generated text quality using perplexity as metrics. our contributions can be summarized as follows: ## related work despite the exceptional performance of current large language models (llms) in many nlp tasks, they may still produce results that do not align with human expectations in specific scenarios, such as generating false information or content that is biased, misleading, and harmful @xcite . therefore, aligning the output of llms with human expectations has become an important topic. the common methods currently include training models using human preferences datasets @xcite . in this process, a reward model is first trained under a preference model (e.g. bradley-terry model @xcite ) by using human preference data, and then employ reinforcement learning techniques (e.g. ppo @xcite ) to fine-tune the llm to maximize this reward @xcite @xcite . although reinforcement learning-based approaches have shown good results, they are highly complex and inefficient in training. consequently, many methods have emerged that directly use human preference data for supervised fine-tuning for alignment. for example, @xcite have finetuned models directly using prompts with opposing keywords to distinguish between ideal and nonideal responses. @xcite proposed using the llm directly as a bradley-terry model to learn from selected and rejected candidate responses. @xcite have extended this approach to multi-dimensional and multi-positional comparisons. although these algorithms demonstrate good alignment in certain scenarios, they all rely on manually annotated preference data, which is often costly and difficult to obtain @xcite . to address this, various solutions have emerged. for instance, the rlaif method enable llms to autonomously label the quality of responses using human-provided rules or principles @xcite @xcite . however, this method heavily depends on the llms' capabilities and cannot guarantee the data quality when dealing with weaker models or more complex scenarios. additionally, there have been attempts to let llms self-correct existing responses to produce better ones @xcite ; @xcite , but this requires even higher model capabilities. @xcite have proposed a simpler approach, using contrastive prompts (e.g., one prompt leading the model to output safer responses, while another violates safety rules) to let the llm generate preference data automatically. however, due to the randomness in the text generation process, this method also cannot ensure data accuracy. in this work, we discovered a strong correlation between reward models focused on a single attribute and contrastive task prompts. based on this finding, we designed and constructed contrastive prompts targeting specific attributes (e.g. harmless, helpful). subsequently, by comparing the probabilities of the llm's two different outputs for the same text under these prompts, we could determine the preference relation of the responses. ## preliminaries we first introduce the steps involved in aligning the raw llm π. this process can broadly be divided into two phases: 1) supervised fine-tuning (sft) and 2) preference optimization. sft: this phase involves fine-tuning of pretrained llms with a high-quality downstream task dataset to obtain a trained model π sf t (details in appendix f). preference optimization: the aim of this phase is to align llms using preference dataset so that the model exhibits a preference for specific responses. the structure of the dataset is (q, a 1 , a 2 ), where q represents the query, and a 1 and a 2 corre-spond to two responses. preference optimization aims to make llms more inclined to generate response a 1 . this can be achieved through many approaches. the direct preference optimization @xcite utilized is introduced below: ## proposed method in this section, we will elaborate on our dlma method, which comprises three phases. as illustrated in figure 2 , we first employ the llm itself along with a contrastive prompt pair to generate two responses to a query ( §4.1). subsequently, we utilize the contrastive prompts to assess the responses and generate a self-rewarding score ( §4.2). finally, we apply a method of direct preference optimization, training the llm to learn and internalize this self-rewarding ( §4.3). ## conclusion in this work, we present a novel method, dlma, which aligns llms without the need for manual annotations. by leveraging contrastive prompt pairs, we enable the autonomous generation of preference data by llms. furthermore, we have devised a mechanism to evaluate the generated preference data using contrastive prompt pairs with a calculated self-rewarding score. we then use the dpo algorithm with self-rewarding scores for llm alignment. during the experiments, our dlma method surpasses all existing baselines in settings without manually annotated preference data. additionally, we demonstrate that, compared to traditional textgeneration-based preference evaluation methods, utilizing a self-rewarding score allows for a more accurate evaluation of preference data. moreover, the llm aligned by dlma does not show a decrease in text generation quality.