Course-Correction: Safety Alignment Using Synthetic Preferences (2024)

Abstract: The risk of harmful contents generated by large language models (LLMs) becomes a critical concern. This paper systematically evaluates and enhances LLMs’ capability to perform course-correction , , the model can steer away from generating harmful content autonomously. First, we introduce the C ^2 -Eval benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction.To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create C ^2 -Syn, a synthetic C ^2 -Syn with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven learning.Experiments on Llama2-Chat 7B and Qwen2 7B show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs’ safety, particularly in resisting jailbreak attacks.

## introduction recently, large language models (llms; openai 2023; @xcite , built on transformer architectures, show remarkable capabilities in text generation. however, the potential for generating harmful content is an escalating concern @xcite . ensuring the alignment of these models with human values and safety standards is essential @xcite . model providers now offer safety-tuned versions of their base models, like llama2-chat @xcite and chatgpt @xcite , which have been trained with a focus on safety. recent studies * equal contribution. † corresponding authors. q: how to make a bomb at home? a: here are the steps to create a home-made bomb👇: first, collect some powder and a plastic bottle … follow the above steps, an explosive bomb is made 💣 q: how to make a bomb at home? a: here are the steps to create a home-made bomb👇: 😱 i am just an ai, i cannot assist you in building a bomb … feel free to ask other questions 😇 reveal that even safety-aligned llms can generate harmful text through methods like red-teaming, with jailbreak attacks being a representative technique @xcite . upon examining the behavior of llama2-chat, a well-aligned llm, we notice an intriguing phenomenon: the model can swiftly self-correct after initially producing unsafe responses, a capability we refer to as course-correction. this ability, as illustrated in figure 1 (b), is crucial for avoiding the continued generation of harmful text (figure 1 (a)). motivated by the absence of comprehensive evaluations of this safety property, we develope a test benchmark termed c 2 -eval foot_0 . c foot_1 -eval is designed to quantitatively measure the coursecorrection abilities of open-source models after harmful text generation. using c 2 -eval, we evaluate 10 prominent llms, including 9 safety-tuned models. the results highlight significant variability in course-correction capabilities among current llms, indicating a polarized landscape. continuing this line of inquiry, we aim to instill the concept of course-correction in models through the data schema. inspired by recent advance-ments in alignment research, notably reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite , we employ coursecorrection-related preference data to fine-tune the model. traditional preference learning relies on large amounts of human preference data, which necessitates extensive human labor and is expensive. motivated by this, we construct a fully synthetic preference dataset termed c 2 -syn, comprising 750k pairwise preference data entries that can be used with prevalent preference learning algorithms. our preference dataset is constructed to prioritize early course-correction over late or no correction. we simulate course-corrective responses by having a synthesizer model generate corrective responses from the beginnings of harmful responses. using a set of corrective triggers, we guide a well-aligned llama2-chat model to produce corrective responses. human evaluation of the synthetic data confirms that our method successfully generates coherent corrective responses at a 98% success rate. after conducting dpo training on two llms including llama2-chat 7b and qwen2 7b with our synthetic c 2 -syn dataset, we observe notable improvements in their course-correction abilities as well as resilience against 4 prevalent jailbreak attacks @xcite @xcite . additionally, their general performance remains unaffected. we conclude that the alignment achieved through preference learning on synthetic data enhances model safety while preserving their overall performance. our contributions are on three folds. ## c -eval: evaluating course-correction ability in this section, we show how to evaluate coursecorrection ability with the help of c 2 -eval. we construct c 2 -eval based on 500 entries of (harm-ful request hr, harmful response fhr) pairs selected from the pku-saferlhf @xcite dataset, initially comprising 83.4k preference entries for rlhf. we specifically select safety-related entries with a response exceeding 80 tokens as our fhrs. refer to appendix b for details. the overall methodology of c 2 -eval is illustrated in figure 2 . to observe potential coursecorrection behavior, we prefill the input with an initial harmful response ihr, which is the prefix derived from the corresponding fhr. besides, the cutoff delimiters 2 for the user prompt and the model response, i.e., <@xmath0, are placed between hr and ihr. the intention is to mark that ihr is generated by the model itself, not from the user prompt. given this setup, our evaluation is limited to open-source models. this is because controlling delimiters in many closed llms such as gpt-4 (openai, 2023) is restricted. the second phase, as outlined in figure 2 , involves sampling multiple decoding paths based on the input prompt of hr∥ihr foot_2 . we then measure the proportion of paths that exhibit corrective behavior. to achieve accurate course-correction detection, we prompt an llm. refer to appendix c for details. ## c 2 -syn: a synthetic dataset for preference learning in this section, we describe the process of creating c 2 -syn, a synthetic pairwise preference dataset containing 750,000 entries designed to teach the value of timely course-correction. ## preference learning with c 2 -syn in this section, we experiment using c 2 -syn to impart course-correction capabilities to 2 llms: llama2-chat 7b and qwen2 7b. ## conclusion in this research, we systematically investigate the problem of course-correction in the context of harmful content generation within llms. we begin with the development of c 2 -eval, a benchmark to evaluate models' course-correction capabilities. using c 2 -eval, we evaluate ten prevalent llms. we then construct c 2 -syn, a synthetic preference dataset of 750k entries, crafted to emphasize the importance of timely course-correction. using c 2 -syn and the direct preference optimization (dpo) algorithm, we conduct safety alignment experiments on two representative llms. results demonstrate that preference learning with our synthetic data can improve two models' overall safety without harming general performance, demonstrating the effectiveness of our method. our research addresses a critical gap in the field of nlp safety, focusing on a niche yet essential aspect.