Automatic Pair Construction for Contrastive Post-training (2024)

Abstract: Alignment serves as an important step to steer large language models (LLMs) towards human preferences. In this paper, we propose an automatic way to construct contrastive data for LLM, using preference pairs from multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the contrastive techniques of SLiC and DPO to SFT baselines and find that DPO provides a step-function improvement even after continuing SFT saturates. We also explore a data curriculum learning scheme for contrastive post-training, which starts by learning from “easier” pairs and transitioning to “harder” ones, which further improves alignment. Finally, we scale up our experiments to train with more data and larger models like Orca. Remarkably, our automatic contrastive post-training further improves the performance of Orca, already a state-of-the-art instruction learning model tuned with GPT-4 outputs, to outperform ChatGPT.

## introduction the rapid evolution of large language models (llms) has ushered in a new era of natural language processing capabilities. these models, when scaled to billions of parameters and pretrained over trillions of text tokens, demonstrate unprecedented proficiency in a wide array of tasks @xcite . various post-training procedures like supervised instruction tuning and reinforcement learning from human feedback (rlhf) fine-tune pretrained llms to better align with human expectations and preferences @xcite @xcite . this additional alignment procedure is crucial, because the pretraining objective of essentially predicting the next token in a text sequence is known to produce llms whose outputs are at times incorrect, irrelevant, or unsafe @xcite . ## related works improving downstream performance of large language models (llms) and aligning them with user preference and designed intents are important to deployment and applications. this can be achieved by fine-tuning these models on responses written by humans or generated with human-written labels and templates. previous works have applied supervised fine-tuning (sft) on both instruction data @xcite @xcite @xcite and dialogue data @xcite @xcite . although sft can successfully adapt an llm to instruction learning or chatting, the model can be further improved by post-training @xcite to meet human preference. a straightforward solution to optimize the human preference is to use reinforcement learning. reinforcement learning with human feedback @xcite first trains a bradley-terry reward model @xcite on human-labeled preference pairs. then, it samples output from the model and scores the output with the reward model. a reinforcement learning algorithm, such as proximal policy optimization (ppo, @xcite is used to optimize the language model for better rewards. rlhf has seen successful applications in downstream tasks @xcite . however, rlhf methods are infamous for their instability, inefficiency, reward misgeneralization and hacking @xcite . recently, there are studies proposing methods for post-training without reinforcement learning. these methods optimize human preference with human-labeled contrastive pairs. besides rlhf, dpo and slic, feedme (openai, 2022) samples model output multiple times and fine-tunes on the best response picked by human labelers. rank responses to align human feedback @xcite adds a ranking loss to the sft loss. the ranking loss promotes responses based on preference ranked by humans or a reward model. human preference is expensive to collect thus difficult to scale up. recently, there have been attempts to automate post-training by replacing the human preference data with model-generated feedback. self-distillation with feedback (sdf, @xcite samples multiple outputs from the model and prompts chatgpt to pick the best response for fine-tuning the model. rl from ai feedback (rlaif, @xcite uses an offthe-shelf llm to replace human labels in the standard rlhf. following that, reinforcement learning from contrast distillation @xcite constructs model-generated contrastive pairs by prompting an off-the-shelf llm to act differently on certain properties, e.g., harmlessness and helpfulness. different from these works, our approach is an offline algorithm, which does not require timeconsuming sampling during training. our approach does not require training a reward model and can be easily scaled up. ## preliminaries reinforcement learning from human feedback (rlhf) to optimize the human preference with reinforcement learning, we need to first train a reward model r τ (x, y) that outputs a reward for a given prompt x and the llm output y. when training the target model, rlhf @xcite uses a reinforcement learning algorithm (usually ppo, @xcite to optimize the reward of a sampled output y from the target model p θ . to regularize the optmization and prevent model degeneration, a kl penalty term between the sequences of distributions over tokens of the target model and a reference model (e.g., sft model) is added to the reward @xcite . this prevents the rl policy from deviating substantially away from the reference model, which often leads to incoherent text output @xcite . sequence likelihood calibration (slic) in contrast to rlhf, slic can exploit pairwise human feedback data and train offline (i.e., without sampling from the target model each time). slic takes a positive example y + , a negative example y -and a reference output y ref from the sft model. in essence, slic encourages the target lm to output sequences those resemble the positive sequence and penalizes those that resemble the negative sequence, while using the reference sequence from the sft model for regularization. the loss function for slic is: direct preference optimization (dpo) similar to slic, dpo is an offline preference optimization method. dpo takes a pair of (pre-computed) positive and negative examples and optimizes the difference between the target model and the reference model (i.e., sft model), which increases the likelihood of the positive example and decreases the likelihood of the negative example. the loss function of dpo is shown below: ## contrastive post-training over pairwise data curriculum contrastive post-training contrastive posttraining involves the construction of positive y + and negative y -sequences in response to the same input x. under the traditional settings of humanfeedback, it is often the case that for some (y 1 , y 2 ) ∼ p (x) sampled from the same llm, human annotators provide a preference as to which is the positive. as this process is expensive, to reduce costs, recent studies @xcite @xcite have investigated the use of pre-aligned models as substitutes for human annotators in providing feedback for post-training methods. however, annotating preference pairs using the largest models, such as gpt-4, on datasets with millions of examples -like the 5m examples used by orca @xcite -would incur a cost of $150k just for calling the api, making it prohibitively expensive as well. in our setting, we choose to sample y + directly from a "superior" llm, y + ∼ p sup , and y -from an inferior p inf . we define one model to be superior to another p sup ≻ p inf if in expectation humans would prefer y + over y -given a reasonable input x. relying on results in tried-and-tested benchmarks @xcite @xcite such as alpaca eval (shown in we acknowledge that there could be many reasons why humans would prefer y + , as previous studies have found that a single reward function may not be sufficient to capture the range of human preferences @xcite . other studies emphasize only a certain property in the contrastive pair, such as helpfulness or harmlessness @xcite . ## experiments using automatically contruscted contrastive pairs, we compare offline contrastive post-training algorithms, slic and dpo, and an online rl method, rlaif, to sft. since both alpaca eval and wiz-ardlm evaluations are pairwise, we choose two ## conclusion and future work in this paper, we propose a setting for automatically constructing data for contrastive post-training large language models. we explore the best method, data and curriculum settings to facilitate automatic post-training. our large-scale experiments with a state-of-the-art model orca further verify the effectiveness of our approach and suggest its potential for improving performance of llms at scale. for future work, we plan to explore both how to better select meaningful contrastive pairs from fixed data regime, and subsequently to continually learning evolving a model with pairs populated by sampling from the model itself at various points through training.