Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models (2024)

Abstract: Data are crucial element in large language model (LLM) alignment. Recent studies have explored using LLMs for efficient data collection. However, LLM-generated data often suffers from quality issues, with underrepresented or absent aspects and low-quality datapoints. To address these problems, we propose Data Advisor, an enhanced LLM-based method for generating data that takes into account the characteristics of the desired dataset. Starting from a set of pre-defined principles in hand, Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. Data Advisor can be easily integrated into existing data generation methods to enhance data quality and coverage. Experiments on safety alignment of three representative LLMs (i.e., Mistral, Llama2, and Falcon) demonstrate the effectiveness of Data Advisor in enhancing model safety against various fine-grained safety issues without sacrificing model utility.

## introduction data serves as a crucial element in the alignment of large language models (llms), as data quality and coverage profoundly impact the utility and safety of llms @xcite @xcite @xcite . since human annotation is costly and does not scale easily, recent studies have utilized llms to produce new datasets @xcite @xcite @xcite , with the main human involvement being the provision of a small set of seed data as in-context examples. although llm-generated data can readily scale, it often suffers from known quality issues @xcite @xcite . pre-vious methods typically generate new data via incontext learning @xcite , without considering dataset-level properties (e.g., coverage and diversity). without additional guidance, the data generator is unaware of the overall dataset statistics, which can lead to the omission of specific aspects and the amplification of its own biases over iterations @xcite @xcite . thus, the generated data can fail to align llms with diverse goals, such as addressing fine-grained safety issues @xcite @xcite . moreover, some issues can manifest as low-quality datapoints, such as ambiguous or redundant questions. although filtering out and refining low-quality data is possible @xcite @xcite , the postprocessing pipelines lead to a notable reduction in preserved data. for instance, alpagasus @xcite noted that 83% of alpaca @xcite data should be discarded due to its detrimental impact on llm alignment. these observations underscore the significance of proactively generating expected data, a direction that remains under-explored in existing literature. in this paper, we propose data advisor, which enhances llm-based data generation by dynamically and proactively incorporating guiding principles of the target dataset (for safety alignment). foot_0 data advisor instructs the data generator to create alignment data with predefined principles, involving both quality and directional control of an independent prompt, as well as the overall statistics of the dataset. with a set of principles in hand, data advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. at the monitor stage, it summarizes the current dataset iteratively, with the last data summary and the newly generated instance as input. at the advise stage, it identifies the current data weaknesses based on the summary, which is sent to the data generator later to guide the generation of the next instance. data advisor can be easily integrated into existing data generation methods, such as self-instruct @xcite , to enhance data quality and coverage. to verify the effectiveness of data advisor, we conduct experiments on the safety alignment of llms. one of the primary challenges in safety alignment is ensuring comprehensive coverage of diverse safety issues @xcite . to address this, data advisor prioritizes the coverage of safety issues, guiding the data generator to produce data that targets missing or underrepresented safety concerns in each iteration. we generated 10k safety alignment datapoints using data advisor, encompassing a wide range of fine-grained safety issues. by integrating the generated data with additional instruction tuning datasets, such as alpagasus, we create a balanced training set. we then train three base llms (i.e., mistral, llama2, and falcon) using this mixture of safety alignment and instruction tuning data. the aligned models demonstrate improved safety across diverse issues without compromising overall model utility compared with the predominant data generation methods like self-instruct. our contributions are three fold. first, we propose data advisor, an llm-based data generation method that dynamically and proactively incorporates the guiding principles of the target dataset. equipped with dataset-level guidelines, data advisor achieves improved data quality and coverage, thereby enhancing llm alignment. second, we demonstrate the effectiveness of data advisor in improving safety alignment without compromising overall model utility. third, we release the generated safety alignment dataset, which covers a wide range of fine-grained safety issues, to support future research. ## preliminaries llms have demonstrated advanced capabilities in instruction following @xcite and incontext learning @xcite . building upon these capabilities, recent studies have applied llms to generate data automatically for further training themselves or other llms, reducing the need for extensive human annotation @xcite . as shown in the bottom of fig. 1 , the typical data generation process begins with a set of seed data serving as the exemplar pool. this process is performed iteratively. in each iteration, the data generator (i.e., an llm) samples multiple exemplars from the pool. these exemplars are then filled into a prompt template and sent to the data generator to produce new data via in-context learning. the newly generated data is subsequently added back to the exemplar pool, marking the end of one iteration. the final dataset is used to train the target llm, enhancing its capabilities. self-instruct @xcite ) is one of the prominent llm-based data generation methods. it uses the target llm itself as the data generator, generating paired prompts and responses in each iteration. in the context of safety alignment, prompts for training should cover diverse safety issues, while responses require careful safety consideration. thus, following @xcite , we generate prompts and responses separately to meet their distinct requirements. specifically, we use an independent safety-aligned llm to provide safe responses to the generated prompts. as another general setting in this paper, we assume that the target llm is unknown in order to demonstrate the generalizability of the generated data. therefore, we use an independent llm as the data generator and validate the effectiveness of the generated data on different target llms. for simplicity, we retain the name "self-instruct" for the baseline throughout the rest of the paper. in the typical data generation process described above, while the llms used as data generators play a crucial role in the quality of individual prompts and responses, they have limited control over the overall data generation process. the properties of the generated data are primarily determined by the initial seed data and the prompts used for data generation. without additional guidance, the data generator is unaware of the overall dataset statistics, can overlook important data properties, and may produce unsatisfactory generated data. ## data advisor data advisor (fig. 1 ) seeks to enhance llmbased data generation methods by dynamically guiding the process with principles aligned to the desired dataset. with an llm acting as the advisor, the advice for data generation is achieved through a series of automatic communications between the advisor and the existing data. with a set of guiding principles, data advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. these principles for data generation specify the purpose of the dataset, key properties to focus on, and additional requirements throughout the generation process. these principles are in the same spirit as collecting human supervision based on a set of guidelines to govern ai behavior, akin to the concept of constitutional ai @xcite . they can vary depending on the application scenarios. we leave further discussion of data generation principles in different scenarios to applied researchers and legal experts. in the following paragraphs, we use diversity and coverage of safety issues as example principles to introduce the details of the method. ## experiment in this section, we first introduce the experimental setup ( §4.1), with a particular focus on the evaluation of safety and utility. this is followed by the presentation of the main results on three representative llms ( §4.2). finally, we provide detailed analyses of fine-grained model performance, data diversity, data mixture, and qualitative results ( §4.3). ## related work in this section, we briefly review two relevant research directions. ## conclusion in this paper, we propose data advisor, an llm-based data generation method dynamically and proactivelyguiding the process with principles aligned to the target dataset. with a set of predefined principles in hand, data advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. experiments on safety alignment of three representative llms demonstrate the effectiveness of data advisor in enhancing model safety against various fine-grained safety issues without sacrificing model utility. further analyses show that data advisor exhibits better data diversity than self-instruct, and its ability to identify dataset weaknesses does not degrade with iterations of data generation. future work can extend data advisor to other scenarios, such as mitigating backdoor in instruction tuning data @xcite , preventing data bias in preference optimization @xcite , and integrating constraints for task adaptation @xcite .