Safer-Instruct: Aligning Language Models with Automated Preference Data (2024)

Abstract: Reinforcement learning from human feedback (RLHF) is a vital strategy for enhancing model capability in language models. However, annotating preference data for RLHF is a resource-intensive and creativity-demanding process, while existing automatic generation methods face limitations in data diversity and quality. In response, we present Safer-Instruct, a novel pipeline for automatically constructing large-scale preference data. Our approach leverages reversed instruction tuning, instruction induction, and expert model evaluation to efficiently generate high-quality preference data without human annotators. To verify the effectiveness of Safer-Instruct, we apply the pipeline to construct a safety preference dataset as a case study. Finetuning an Alpaca model on this synthetic dataset not only demonstrates improved harmlessness but also outperforms models fine-tuned on human-annotated safety preference data, all the while maintaining a competitive edge in downstream tasks. Importantly, our Safer-Instruct framework is versatile and can be applied to generate preference data across various domains, extending its utility beyond safety preferences. It addresses the challenges in preference data acquisition and advances the development of more capable and responsible AI systems. For dataset and code implementation, see https://github.com/uscnlp-lime/safer-instruct/.

## introduction reinforcement learning from human feedback (rlhf) has proven to be an effective strategy in enhancing model capability and mitigating harmful outputs generated by language models @xcite . by fine-tuning models on preference data through rlhf, we can provide explicit guidance on what constitutes appropriate and responsible language use. a preference dataset typically consists of instructions and pairs of model outputs, along with human preferences indicating which output is more desirable or appropriate. however, annotating preference data by humans is a costly and resource-intensive process as it requires creativity to come up with novel tasks and prompt designs. annotators must not only craft innovative jailbreak prompts but also provide both preferred and dispreferred responses to these prompts @xcite . additionally, while there has been promising research on automatically generating instruction data by querying expert models like gpt-4 (openai, 2023), they rely on either a limited set of manually crafted scenarios @xcite or the expansion of manually composed seed instructions @xcite @xcite . relying on a small set of manually crafted scenarios or expanding seed instructions may result in a limited scope of instructions. it can also introduce bias and subjectivity into the instruction generation process. furthermore, rlhf requires both preferred and dispreferred responses, but expert models like gpt-4 have robust safety filters that prevent the generation of harmful content, which makes it difficult to create dispreferred responses and malicious instructions for safety preference datasets. these challenges underscore the complexities involved in acquiring the necessary data for effective preference training, which significantly hurdles the development of safer and more responsible ai systems. in this work, we introduce safer-instruct, an automated process for constructing large-scale preference data (see figure 1 ). firstly, we employ reversed instruction tuning to train a model that can do instruction induction: generating instructions based on responses. then, through instruction induction, we efficiently generate instruction the three primary colors are red, blue, and yellow. ## related work preference training. a series of works have found evidence that rlhf can significantly improve model performance across various natural language processing tasks, such as text summarization @xcite and mitigating harmful effects @xcite . at a high level, this process involves modeling human preferences using a reward function, which is subsequently used to train language models through rl methods such as proximal policy optimization (ppo) @xcite . an alternative approach to rlhf for preference training is direct preference optimization (dpo) @xcite , which implicitly optimizes the same objective as existing rlhf algorithms but is simple to implement and straightforward to train. preference dataset. general-purpose preference datasets are typically human-annotated @xcite . human annotators are typically asked to interact with some language models and rank the generated responses. the responses sometimes can also be ranked by an expert ai system @xcite . some preference datasets are collected in the wild from online forums such as stackexchange @xcite , in which the number of votes a comment receives is used as reward signals. however, such preference signals are heavily influenced by the majority opinion, and the questions from online forums are usually different from the user's queries to language models. in addition to general-purpose preference datasets, there are also preference datasets built for model safety. anthropic's helpfulness and harmless (hh) @xcite and red teaming dataset @xcite are human-annotated safety preference datasets where crowd workers write a chat message to some unknown models, and are asked to choose the more helpful and honest response from two response candidates. typically, the response options are directly generated by the models themselves, although occasionally they may also include self-revised versions of the initial model-generated response @xcite . beaver tail @xcite is another humanannotated safety preference dataset collected similarly to anthropic's data, but the instructions are selected from the red teaming dataset, and the responses are generated by an alpaca model. however, annotating preference datasets is expensive. the cost of the crowd workers alone to annotate anthropic's red teaming data (around 40k instances) is at least $60k @xcite , indicating an urgent need for a better data acquisition method. ## safer-instruct framework annotating large-scale instruction data is challenging, as it requires creativity for novel tasks and prompts. safety preference data for rlhf is even more complex, with a demand for innovative jailbreak prompts and both preferred and dispreferred responses. existing methods often rely on limited manually crafted scenarios, which could result in a limited scope and complexity of instructions @xcite . to address these issues, we introduce safer-instruct, a pipeline for autonomously constructing preference datasets without human annotators. the pipeline is depicted in figure 1 . ## safer-instruct data in this section, we apply our method to construct the safer-instruct (si) dataset, a safety preference dataset for llms. the statistics of the dataset are shown in ## experiment we demonstrate the efficacy of our pipeline by finetuning an alpaca model @xcite on the si dataset we constructed. we apply the direct preference optimization (dpo) algorithm @xcite to train our model. after training, we evaluate our model on helpfulness, harmlessness, ## conclusion in conclusion, we introduce safer-instruct, a groundbreaking pipeline that addresses the challenges of constructing large-scale preference data for rlhf. our approach leverages reversed instruction tuning, instruction induction, and expert model evaluation to autonomously generate highquality preference data without relying on resourceintensive human annotation. by applying safer-instruct to train language models, we significantly improve their harmlessness while maintaining competitive performance in conversation and downstream tasks without the requirement of human annotations. crucially, our framework is adaptable and can be employed to generate preference data across a wide range of domains, extending beyond the safety preference dataset. this research not only drives the advancement of more capable and responsible ai systems but also provides a valuable resource for evaluating model safety. reinforce llm bias. the safer-instruct process relies upon an expert model to generate and evaluate responses. this could potentially inherit and amplify the biases exhibited in the expert model. the exact definition of unsafe can also be subjective, and some potential safety issues may not be captured by our filtering process.