Preference-Guided Reflective Sampling for Aligning Language Models (2024)

Abstract: Iterative data generation and model re-training can effectively align large language models (LLMs) to human preferences. The process of data sampling is crucial, as it significantly influences the success of policy improvement. Repeated random sampling is a widely used method that independently queries the model multiple times to generate outputs. In this work, we propose a more effective sampling method, named Preference-Guided Reflective Sampling (PRS). Unlike random sampling, PRS employs a tree-based generation framework to enable more efficient sampling. It leverages adaptive self-refinement techniques to better explore the sampling space. By specifying user preferences in natural language, PRS can further optimize response generation according to these preferences. As a result, PRS can align models to diverse user preferences. Our experiments demonstrate that PRS generates higher-quality responses with significantly higher rewards. On AlpacaEval and Arena-Hard, PRS substantially outperforms repeated random sampling in best-of- N sampling. Moreover, PRS shows strong performance when applied in iterative offline RL training.

## introduction large language models (llms) have made significant advances @xcite @xcite . these models are typically aligned with human expectations through finetuning. this is achieved by using reinforcement learning from human feedback (rlhf), which mitigates the generation of harmful, biased, or irrelevant outputs @xcite . both online and offline rl methods have been explored for rlhf @xcite @xcite . iterative offline training provides a more efficient alternative than online training, by allowing outputs to be pre-generated and reused to facilitate iterative improvements in policy. effective data sampling is crucial for iterative model re-training, as it directly influences the effectiveness of the resulting policy @xcite . repeated random sampling (as shown in fig. 2 ) is an effective method and has been widely used for data generation in previous work @xcite . it independently calls the model multiple times to get samples. then higher-quality data will be maintained to update the policy model. however, the vast output space compromises its efficiency since the inherent randomness may result in inefficient exploration in the sampling space. also, the simple generation strategy cannot learn from and adapt dynamically based on previously generated samples. furthermore, with only the supervision of the reward model, it is hard to optimize the outputs to align to diverse and personalized preferences. we propose a new sampling method named preference-guided reflective sampling (prs) to improve data generation. different from random figure 2 : comparison of repeated random sampling and our method prs. prs adopts a tree-based generation framework that learns to adapt and adjust its outputs by reflecting on its already generated data. it can incorporate a specific user preference to optimize responses that align with it. adjusting preferences will generate tailored responses. for random sampling, it generates samples independently and can use the best-of-n (bon) method to find the best sample. both methods share the same sampling budget, which samples the same number of responses for each prompt. we study preference-controlled text generation for the task of instruction following and keywordfocused document summarization. in our experiments, we first evaluate prs against various baselines in generating training samples with diverse policy models ( § 5.1). in § 5.2, we investigate its application for aligning llms to adhere to explicit preferences provided in the inputs using offline rl training. we further explore preference adaptation, toxicity reduction, and other areas in § 5.3. our contributions in this work are as follows: ## related work offline rl offers an efficient alternative to online rl @xcite . @xcite , @xcite , and @xcite emphasize data generation and model refinement. repeated random sampling is a simple but effective method for data generation. brown et al. (2024) demonstrate that scaling inference compute can significantly improve the model performance in problem solving. bai et al. (2022b) leverage the llm's reflection capacity to continuously refine model responses. however, they only focus on harmless responses, whereas our work is applicable across a broad spectrum of preferences. moreover, different from ours, their work does not aim to improve data sampling for rl training. feng et al. (2023) use monte carlo tree search (mcts) with token-level rewards, but ours employs sequencelevel rewards based on cost-effective tree-based generation, with input preferences to guide the generation. scheurer et al. (2023) advocate for training models using human language feedback, but we employ the model itself to generate language feedback. a more detailed discussion of the related work is in appendix a. the sample with highest reward feedback generation go deeper (optional) what type of visual aid is the best option for depicting a timeline? question i prefer the ai model to provide concise and accurate responses that are supported by reliable sources or references. preference 1. timeline chart: this is a graphical representation of events or milestones in chronological order. ... 2. ... * provide references or sources to support each claim made in the response ... * break down the response into smaller, more manageable sections ... 1. timeline chart: this is a graphical representation of events or milestones in chronological order. ... 2. gantt chart: a gantt chart is a type of bar chart used to show a schedule of a ... references * [1]: wikipedia, "timeline," <https ://en.wikipedia.org/wiki/@xmath0... response feedback refined response 2. sample response 3. provide feedback 4. revise response 1. add explicit preference (a) reflective refinement (b) tree-based generation 3 preliminaries offline rl. rlhf utilizes human feedback to finetune a pre-trained llm with human preferences. the preference human feedback can be utilized to train a reward model r(x, y), given an input x and a response y. following gülçehre et al. (2023), we employ offline rl to synchronize the llm policy with the trained reward model. this process, beginning with the policy initialized by supervised fine-tuning (sft) on labeled data, involves iterative cycles of data generation and model re-training. the policy of the llm, π θ , parameterized by θ, produces a response y given the input x, i.e., y ∼ π θ (y|x). using the labeled data d 0 , the llm is trained with the negative log-likelihood (nll): then it repeats data generation and model retraining to align the language model: 1. data generation: each iteration leverages the policy from the previous cycle to generate n responses y for each input x in the unseen dataset u k . the reward model, r(x, y), evaluates these responses to generate rewards. best-of-n strategy or a reward threshold is used to identify the high-quality examples. 2. model re-training: the newly generated data, along with all prior data, is used to refine the model in the subsequent re-training phase. ## method we aim to improve the data generation process to enhance offline rl training. we first introduce preference-guided reflective sampling (prs), and then study the task of preference-controlled instruction following using offline rl training. ## conclusion we introduce prs, an improved sampling method designed to enhance iterative model improvement. in contrast to repeated random sampling, prs enables more efficient generation through a tree-based approach. by allowing the specification of preference in the input, prs optimizes responses to better align language models with diverse user preferences. our comprehensive evaluation shows that prs consistently generates higher-quality samples. on alpacaeval and arena-hard, prs significantly outperforms random sampling in the best-of-n setting. additionally, prs excels when applied to iterative offline rl training.