Aligning Large Language Models through Synthetic Feedback (2023)

Abstract: Aligning large language models (LLMs) to human values has become increasingly important as it enables sophisticated steering of LLMs. However, it requires significant human demonstrations and feedback or distillation from proprietary LLMs such as ChatGPT. In this work, we propose a novel alignment learning framework with synthetic feedback not dependent on extensive human annotations and proprietary LLMs. First, we perform reward modeling (RM) with synthetic feedback by contrasting responses from vanilla LLMs with various sizes and prompts. Then, we use the RM to simulate high-quality demonstrations to train a supervised policy and further optimize the model with reinforcement learning. Our resulting model, Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms recent open-sourced models, which are trained on the outputs of InstructGPT or human-annotated demonstrations, in alignment benchmarks. In human evaluation, our model is preferred to Alpaca and Dolly-v2, 55.0% and 58.5% of the time, respectively. Further analyses demonstrate the efficacy and importance of synthetic feedback in our framework.

## introduction alignment learning has been an essential learning scheme to align the behaviors of large language models (llms) with human values like safety and truthfulness while following the intention of users accurately @xcite . vanilla llmsthose not aligned yet -could misunderstand user intentions or produce unsafe and inaccurate responses. desirable human values such as helpfulness, harmlessness, or honesty can be defined, and human demonstrations with these values are then used for the alignment learning @xcite . 1 the code is available at github.com/naver-ai/almost. figure 1 : a procedure of reward modeling through synthetic feedback. we assume that the response from a larger llm with more and better demonstrations might be better overall. we train a reward model with synthetic comparisons generated top on the assumption. typically, alignment learning consists of three stages: supervised fine-tuning (sft), reward modeling (rm), and reinforcement learning from human feedback (rlhf) @xcite . however, the three-stage training recipe requires significant human effort, especially in the first two stages. more specifically, both the sft and rm training stages must be provided with an abundance of high-quality human demonstrations and ranking datasets for obtaining models to facilitate rlhf. for instance, @xcite prepare and utilize 13k human demonstrations and 33k comparisons. on the other hand, self-instruct @xcite attempts to generate synthetic self-generated instruction datasets using in-context learning with a few seed demonstrations. meanwhile, the release of llama @xcite brings upon many open-sourced aligned llms trained on the outputs of proprietary llms or human-annotated instructions. however, it still heavily depends on proprietary llm apis such as instructgpt and chat-gpt @xcite @xcite or intensive human annotations @xcite . in this paper, we introduce a novel framework for alignment learning that only requires minimal human labor and is not dependent on proprietary llms. unlike the conventional alignment learning procedure that collects demonstration first @xcite , we first develop a reward model (rm) on a synthetic comparison dataset constructed by contrasting outputs from vanilla llms in various configurations, as shown in figure 1 . the rules of generating these synthetic ranking data originate from our hypothesis that the responses generated by larger, optimally prompted models are superior to those produced by smaller, inadequately prompted models, as reported by previous work @xcite . then, we introduce a reward-model-guided self-play (rmsp) to simulate high-quality demonstrations with rejection sampling using the rm @xcite . we train llama-7b @xcite on the synthetic demonstrations (sft) and further optimize the model with rewards from the synthetic rm, namely, reinforcement learning from synthetic feedback (rlsf). our aligned language model with synthetic training dataset (almost) outperforms alpaca @xcite -distilled from in-structgpt @xcite and openassistant @xcite that are trained on human-annotated demonstrations in the alignment-related benchmarks @xcite @xcite . notably, our model is preferred to recent open-sourced models, alpaca and dolly-v2, 55-58% of the time (winning rate) in human evaluation without distillation from proprietary llms nor intensive human annotations. we speculate the strong performance of our model is due to the empirical indicators of well-aligned behaviors that have been effectively incorporated into a strong backbone model through synthetic feedback, allow-ing the inherent capability to self-align to elicit and partially replace the need for human feedback. our main contributions are three folds: ## method in this section, we will describe detailed procedures of our framework as depicted in figure 2 . ## evaluating alignment of almost we validate our resulting model, aligned language model with synthetic training dataset (almost), in three alignment benchmarks, static hhh evaluation @xcite , truthfulqa @xcite , and vicuna questions @xcite . vicuna questions we test our models using vicuna evaluation questions @xcite . it consists of 80 questions on various topics spanning general qa, writing, reasoning, etc., to identify user preference foot_6 . first, two different models gen- we ask the workers to choose the more helpful, harmless, and honest answer with careful consideration over the contents in the answers @xcite . please see appendix e.1 for more details of the human evaluation. in addition, we conduct an automatic evaluation using gpt-4. in the test, gpt-4 assesses the two answers by giving a 1-10 scalar score for the corresponding answer and providing appropriate explanations for the judgment @xcite . even though it is not a rigorous evaluation, we can compare the overall responding qualities of the models with reasonable costs. considering the positional bias of the gpt-4 assessment foot_7 , we evaluate the same instance twice by reversing the order of the two answers. ## conclusion in this work, we propose a novel framework for aligning llm with human values by introducing synthetic feedback. we identify better responses from vanilla llms with various sizes and prompts, relying on empirical prior knowledge. we first train a reward model with synthetically generated comparisons. then, we produce another synthetic dataset to train aligned policies using the reward model. experimental results demonstrate the effi-cacy of our framework showing outstanding performances in the alignment benchmarks. we believe the strong performance of our model is derived from the effective incorporation of empirical indicators of well-aligned behaviors through synthetic feedback. furthermore, our method is costeffective in that it does not require extensive human demonstrations and not depend on the proprietary llms.