Self-Criticism: Aligning Large Language Models with their Understanding of Helpfulness, Honesty, and Harmlessness (2023)

Abstract: Recently, there has been a notable surge in the significance of large language models (LLMs) that engage in conversational-style interactions, such as ChatGPT and Claude, as they contribute significantly to the progress of artificial general intelligence (AGI). Typically, these models undergo a two-phase fine-tuning process: instruction fine-tuning (IF) and reinforcement learning from human feedback (RLHF). These methods aim to align the LLMs to be helpful, honest, and harmless (HHH). However, RLHF, which incorporates independent reward models trained on high-quality human feedback datasets, incurs high costs in terms of hardware resources and human efforts. Therefore, we explore the possibility of aligning LLMs with their own understanding of HHH through IF and in-context learning (ICL). In this study, we propose a novel framework called Self-Criticism, which allows LLMs to align themselves with HHH based on the definition they learned from a large-scale text corpus. We begin by employing IF on a given instruction set and learning HHH discrimination through few-shot ICL. Subsequently, the LLMs evaluate their own generated responses and learn to produce “better” responses based on self-judgment. Finally, the model is retrained based on the self-generated responses to distill the whole process. By analyzing our proposed method, we also find interesting connections between Self-Criticism and goal-conditioned reinforcement learning, and pseudo-labeling. Experimental results demonstrate that this method achieves nearly identical performance to RLHF in terms of both human evaluation and evaluation by other LLMs, with only a minimal alignment tax.

## introduction in recent times, large language models (llms) @xcite have * equal contributions. in the second phase, a reward model is trained to incorporate and learn from human feedback based on the human-labeled output generated by the model in the first phase. finally, reinforcement learning is applied to enable the llms to achieve high rewards evaluated by the reward models. the last two steps are commonly recognized as reinforcement learning from human feedback (rlhf) @xcite . several models and services, such as chatgpt @xcite and claude, have demonstrated remarkable performance by undergoing the aforementioned three training phases. the incorporation of rlhf techniques has been recognized as crucial in infusing human values into these models. nevertheless, implementing rlhf on llms presents challenges. it necessitates the development of a reward function, which relies on a substantial amount of human-labeled data and may be susceptible to misalignment. additionally, optimizing reinforcement learning algorithms poses difficulties, particularly when used in parallel llms training with dif-ferent distributed frameworks (i.e., megatron and deepspeed) @xcite while carefully managing graphical memory constraints. therefore, incorporating human values into llms through the implementation of rlhf requires significant resources and should be evaluated in terms of cost-effectiveness, especially in industrial applications. since incorporating human value into the llms is to achieve hhh alignment, is it possible to align llms with the concept of hhh by leveraging their own understanding acquired from a largescale text corpus, without using rlhf? this approach seems reasonable as humans have already demonstrated their understanding of helpfulness, harmlessness, and honesty in written form. therefore, in this paper, we introduce a new framework called self-criticism that achieves llm alignment solely through in-context learning (icl) and sft. initially, we employ sft on an instruction set to ensure the model's ability to follow instructions. then, we use carefully crafted prompts for fewshot icl, enabling the model to evaluate its own generated response and improve upon it. finally, we perform sft once again to distill the entire process with the selected response. each component of our proposed method is driven by technical considerations rather than heuristic approaches. to begin, we employ icl @xcite and sft for reward generation, which is effectively employing pseudo-labeling techniques commonly used in semi-supervised settings with limited labeled data. next, our policy generation relies on the model's own judgment, employing a reward-constrained policy maximization approach @xcite . lastly, when we distill the selected action using sft, we engage in best action imitation learning, with the model itself determining the "best" action @xcite @xcite @xcite . in order to comprehensively evaluate our approach, we conduct a thorough comparison between the trained model and models trained by sft and rlhf. this evaluation is performed on a holdout instruction set that encompasses various scales, and the labels are provided by both human annotators and chatgpt @xcite . this evaluation framework has been widely acknowledged in previous studies as a reliable method for assessing the performance of sft. furthermore, we evaluate our method on multiple evaluation benchmarks, specifically examining the impact of alignment tax @xcite . remarkably, our approach achieves performance levels close to those of rlhf, while incurring minimal alignment tax. ## methods many pieces of literature discuss alignment techniques for llms. for a comprehensive review of these works, we invite readers to refer to appendix a. in this work, our objective is to align the model's comprehension of hhh without resorting to reinforcement learning training manner. initially, we apply sft to a given instruction set to ensure that the model can follow the instruction. subsequently, we employ few-shot icl using thoughtfully designed prompts to train the model as an hhh discriminator. finally, we construct a generation prompt that enables the model to generate a "better" response based on its past evaluations. the full framework is shown in figure 1 . to initially ensure the pre-trained llm follows the instruction, we first perform sft based on the causal and decoder-only model p θ with parameter θ. the algorithm of self-criticism is shown in appendix e. ## experiment for model training, we implement the dolly dataset @xcite which contains 15k human written responses with high quality and diverse instruction types. we divide the dataset into two parts, of which 50% is the d sf t for sft, 30% is the d r , and 20% is the test set d t for model evaluation. we select the bloomz model as our base model because these model series compose various scales models which can easily test the scaling effect of our proposed method. bloomz @xcite is a family of pre-trained models which support multilingual language and provide multiple model capacities, which demonstrate excellent ability to follow instructions in many tasks. we follow the standard hyper-parameter reported in @xcite to fine-tune the model. to be specific, the max sequence length is 768, the learning rate is 1e-5 ,and the weight decay is 0.01. the model is trained on inter xeon cpus with 512gb memory and one a100 gpu with 80g graphical memory. to fairly compare our proposed framework, we utilize the sft model based on the instruction set and rlhf model which is further fine-tuned on the dolly dataset @xcite as our baseline methods. the reward model of the rlhf model is trained on the dataset hh-rlhf, oasst1, and wombat which is provided by anthropic, openassistant, and yuan et al. @xcite @xcite . we test these models on the hold-out set d t and generate the response following the nucleus decoding policy with identical decoding parameters. we set the max length as 512, top-p as 0.65, temperature as 0.9, repetition penalty as 1.1, and length penalty as 1.1. since previous work @xcite reported that the perplexity score used in typical nlp tasks is not strictly correlated with the response quality. we follow the method introduced in the @xcite to invite five human annotators to compare the quality between the responses generated by the different models. the human annotators are trained to label excellent, pass, and fail for each output. to ensure the effectiveness of human annotation, we randomly provide the data to each annotator, that the data have been labeled by other annotators, to ensure the inner agreement rate is consistently higher than 90%. if the model outputs don't satisfy the criteria of harmlessness and honesty, they will be labeled as fail directly. then, we evaluate the model with gpt-3.5-turbo with the prompt shown in appendix d. based on the experimental results shown in the section 4 and section 5. we can observe that the model can generate a higher quality response than the sft model and is comparable with the rlhf model. ## conclusion this framework leverages the llm's own comprehension of helpfulness, honesty, and harmlessness, which have been already encoded in pretrained models. within this framework, each learning procedure is supported by techniques from the domains of reinforcement learning and semisupervised learning, rendering the framework both interpretable and feasible. through model generation experiments evaluated by both human assessors and gpt-3.5-turbo, our experimental results demonstrate that our proposed method achieves comparable outcomes to rlhf. furthermore, our ablation study confirms the effectiveness of our framework, as it exhibits minimal alignment tax similar to the rlhf and sft models.