Learning Preference Model for LLM s via Automatic Preference Data Generation (2023)

Abstract: Despite the advanced capacities of the state-of-the-art large language models (LLMs), they suffer from issues of hallucination, stereotype, etc. Preference models play an important role in LLM alignment, yet training preference models predominantly rely on human-annotated data. This reliance limits their versatility and scalability. In this paper, we propose learning the preference model for LLMs via automatic preference data generation (AutoPM). Our approach involves both In-Breadth Data Generation, which elicits pairwise preference data from LLMs following the helpful-honest-harmless (HHH) criteria, and In-Depth Data Generation, which enriches the dataset with responses spanning a wide quality range. With HHH-guided preference data, our approach simultaneously enables the LLMs to learn human preferences and align with human values. Quantitative assessments on five benchmark datasets demonstrate the reliability and potential of AutoPM, pointing out a more general and scalable way to improve LLM performance.

## introduction with the rapid development of the large language models (llms), the state-of-the-art llms (e.g., gpt series @xcite , opt @xcite , bloom @xcite , and llama @xcite ) exhibit remarkable emergent capabilities @xcite and can adeptly generate coherent responses in accordance with user instructions after supervised finetuning (sft) @xcite @xcite . despite their advanced generation capacities, llms may produce inaccurate and potentially detrimental content that is not readily discernible by human users. aligning llms with human values is thus of central consideration before deploying llms to society. governing model behavior from implicit signals underlying human feedback has long been a principled methodology for llm alignment. exemplary work employs preference models to embed those human feedback signals @xcite and recent publications like instructgpt @xcite have made significant progress along this direction. these preference models also constitute a cheap yet reliable model-based evaluation metric @xcite and have thus been widely adopted. nonetheless, the acquisition of the preference model relies heavily on human-annotated preference data. a standard data collection procedure involves obtaining multiple machine-generated responses for each prompt, followed by engaging annotators to rank these responses. to guarantee alignment with human values, recent work @xcite follows hhh criteria @xcite in their annotation process. they encourage crowdworkers to intentionally goad virtual assistants to produce helpful or harmful responses during the interaction, which further intensifies the reliance on manual annotation and impedes the versatility and scalability of preference models. a line of work starts to explore the possibility of leveraging llms themselves for gathering preference data @xcite . however, these methods still adhere to conventional preference data collection pipelines, with the exception of replacing human annotators with @xcite and @xcite . such naive data collection strategies lack a comprehensive guideline and therefore fail to align the preference model well with human values (see section 5). to address these shortcomings, we propose au-topm, an approach for learning the preference model via automatic preference data generation from llms. our guideline-driven data generation method systematically directs the preference data synthesis from performant llms, achieving broad coverage and diversity. moreover, as our data synthesis guideline closely adheres to the hhh criteria @xcite , our approach enables the llms to learn human preferences and simultaneously align themselves with human values. specifically, autopm consists of in-breadth data generation and in-depth data generation. the in-breadth data generation prompts pairwise preference data from llms that either match or violate human standards from various aspects in accordance with the hhh criteria as the guidance of human preferences. the in-depth data generation, on the other hand, employs a novel hhh-guided sequential post-editing process to construct preference data with broad coverage of quality. through this novel paradigm of preference data collection, autopm can effectively generate diverse preference data that adheres closely to human preference guidelines without human annotation. with gpt-3.5 as the main source of synthetic preference data, we amass 42k preference data and develop autopm based on llama @xcite . extensive evaluation on 5 benchmark datasets attests to the superiority of autopm. experimental results show that autopm's predictions are highly consistent with humans and gpt-4. au-topm can even elevate the performance of strong llms. besides, we have two intriguing findings: preference model for llms. a well-learned preference model can significantly contribute to evaluating and aligning llms @xcite @xcite . early work @xcite begins to use preference models as human preference feedback in specific nlp tasks. instruct-gpt @xcite leads the way to learning the preference model for improving general ai assistants. the diversity of the prompts-responses set is a key factor for training preference models. anthropic @xcite asks crowdworkers to have open-ended conversations with ai assistance to cover more helpful and harmless cases. work like shp @xcite collects preference data from real-world websites like reddit. recent work starts leveraging strong llms to collect preference data @xcite , but still follow the conventional preference data collection pipeline except for ranking the responses using more advanced gpt-3.5 and gpt-4. in contrast, the proposed autopm directly generates preference data from llms governed by hhh criteria, ensuring not only a high level of accuracy but also increased diversity. alignment of llms. alignment @xcite of llms aims to build agents that are better at following user intentions. instructgpt @xcite propose to improve llm alignment in two ways: supervised fine-tuning (sft) with demonstration data and reinforcement learning from human feed-helpful honest harmless hhh criteria instantiate instruction: rewrite this sentence to use active voice: "the cake was baked by my grandmother." given the question, generate a useful answer a, and a useless answer b. helpful: (useful, useless), … honest: (correct, contains wrong information), … harmless: (impartial, biased/discriminatory), … random sampling answer a: my grandmother baked the cake. ## automatic preference data generation autopm comprises two components: in-breadth data generation and in-depth data generation. in addition, we also use paired models data to further supplement the data. the detailed descriptions follow below. ## conclusion in this paper, we propose learning the preference model for llms via automatic data generation under the human preference guideline, named au-topm. the proposed autopm contains in-breadth data generation and in-depth data generation, and paired models data generation. we conduct extensive quantitative assessments on 5 benchmark datasets, demonstrating the reliability of autopm and its potential for enhancing llms. we believe that autopm shed light on a more general and scalable way for learning preference models.