Adversarial Preference Optimization: Enhancing Your Alignment via RM - LLM Game (2024)

Abstract: Human preference alignment is essential to improve the interaction quality of large language models (LLMs). Existing alignment methods depend on manually annotated preference data to guide the LLM optimization directions. However, continuously updating LLMs for alignment raises a distribution gap between model-generated samples and human-annotated responses, hindering training effectiveness. To mitigate this issue, previous methods require additional preference annotation on newly generated samples to adapt to the shifted distribution, which consumes a large amount of annotation resources. Targeting more efficient human preference optimization, we propose an Adversarial Preference Optimization (APO) framework, in which the LLM and the reward model update alternatively via a min-max game. Through adversarial training, the reward model can adapt to the shifted generation distribution of the LLM without any additional annotation. With comprehensive experiments, we find the proposed adversarial training framework further enhances existing alignment baselines in terms of LLM helpfulness and harmlessness. The code is at https://github.com/Linear95/APO.

## introduction learned from massive textual data with billions of parameters, large language models (llms), such as gpt-4 (openai, 2023) and gemini @xcite , have shown remarkable ai capabilities, especially in domains of natural language processing @xcite , logical reasoning @xcite , and programming @xcite . among the training techniques that help llms achieve such success, human preference alignment finetunes llms to follow users' feedback, which has been widely rec-* equal contribution. to utilize preference data, current feedback alignment methods are proposed mainly from three perspectives @xcite : reinforcement learning @xcite , contrastive learning @xcite @xcite , and language modeling @xcite @xcite . reinforcement learning with human feedback (rlhf) @xcite is the earliest exploration and has been acknowledged as the mainstream for llm alignment @xcite . rlhf first learns a reward model from the human preference data, then optimizes the expected reward score of the llm's output samples via the proximal policy optimization (ppo) algorithm @xcite . although widely used, rlhf has been criticized as being unstable during the fine-tuning and complicated in implementation and computational resource consumption @xcite . towards more efficient and stable training, instead of directly optimizing the non-differentiable rewards, contrastive learning methods enlarge the likelihood gap between preferred and rejected response pairs @xcite @xcite . alternatively, language modeling-based methods remain using language modeling loss to align preference, but with different data preparation strategies @xcite @xcite . for example, rejection sampling @xcite select responses with top reward scores as the language modeling fine-tuning samples, while @xcite and @xcite add different prompts to different responses based on the corresponding preference levels. although contrastive-learning and languagemodeling-based methods have partially alleviated the inefficiency of rlhf, the sampling distribution shifting problem @xcite still hinders the alignment effectiveness: after a few steps of rlhf updates, a distribution gap emerges between llm generated samples and preferenceannotated data (as in figure 1 ). consequently, the reward model learned with human annotation loses its performance in providing faithful reward signals on newly generated responses, which damages the alignment performance. to address this problem, most aforementioned alignment methods require additional annotation of human feedback on newly generated responses after a certain amount of llm updating steps @xcite , which leads to increasingly massive manpower costs @xcite . besides, the vast time consumption of extra manual annotation also significantly slows down the alignment training process. to reduce the manual annotation efforts and improve the preference optimization efficiency, we propose a novel adversarial learning framework called adversarial preference optimization (apo). inspired by generative adversarial networks (gans) @xcite , we conduct an adversarial game between the reward model (rm) and the llm: the llm generates responses to maximize the expected reward score, while the rm aims to distinguish the score difference between golden and sampled responses. to verify the effectiveness of the apo framework, we conduct experiments on the helpful&harmless @xcite datasets with alpaca @xcite and llama-2 (touvron et al., 2023b) as the base models. with the same amount of human preference data, both the llm and rm receive additional performance gains through the apo game, compared with several commonly used llm alignment baselines. ## preliminary human preference alignment aims to fine-tune the llm response policy π θ (y|x) with a group of human preference data d @xmath0, y w , y l )}, so that the llm can generate more satisfying responses to improve the human-model interaction quality. in each preference triplet (x, y w , y l ), y w ≻ y l means response y w is more "preferred" than y l with respect to input x. to align the llm, a reward model (rm) @xcite ) r ϕ (x, y) is commonly utilized to score the quality of the llm generated samples. rm learns human preferences d p with a ranking loss @xcite with equation 2, training rm with the bradley-terry ranking loss can be explained as the loglikelihood maximization of q ϕ : with a learned rm r ϕ (x, y), human preference alignment methods @xcite @xcite target on maximizing the reward expectation of generated responses: however, ppo suffers from implementation complexity and training instability @xcite . recent studies try to avoid online reinforcement learning with offline schemes. dpo @xcite finds a connection between the reward model and llm's optimal solution, then replaces the reward model with the likelihood ratio of π θ and π ref , as l dpo (π θ ) := analogously, other methods consider human feedback learning from the perspective of contrastive learning. for example, rrhf @xcite propose a ranking loss as l rrhf (π θ ) := generative adversarial networks (gans) are a classical group of unsupervised machine learning approaches that can fit complicated real-data distributions in an adversarial learning scheme @xcite . gans use a discriminator d(•) and a generator g(•) to play a min-max game. the generator tries to cheat the discriminator with reallooking generated samples, while the discriminator aims to distinguish the true data and the samples: in policy optimization of reinforcement learning, inspired by gans, @xcite propose generative adversarial imitation learning (gail): in natural language generation, gans have also been empirically explored @xcite , where a text generator samples reallooking text and a discriminator makes judgment between the ground-truth text and generated samples. textgail @xcite applies gail (equation 10) into text generation, which optimizes the language model as a response policy π θ (y|x), by reducing the distribution divergence between model-generated samples and human responses. ## adversarial preference optimization we begin with a revisit of the human preference alignment in a mathematical optimization form: based on equation 12, we conduct an adversarial game, in which llm π θ (y|x) needs to improve its response quality to get a higher expected reward, while rm r ϕ (x, y) tries to enlarge the reward gap between the golden responses and the generation from π θ (y|x). inspired by the original preference alignment objective (equation 11), we add two kl regularizers to π θ and r ϕ respectively to prevent over-fitting and degeneration. here p (y ≻ ỹ|x) denotes the ground-truth human preference probability, and q ϕ (y ≻ ỹ|x) is described in equation 2. note that we use the reverse kl[π θ ∥π ref ] to constrain the generative model π θ but the forward kl[p ∥q ϕ ] for the discriminate model r ϕ . our intuition is that kl[π θ ∥π ref ] can be estimated with π θ -generated samples, paying more attention to the generation quality; while kl[p ∥q ϕ ] is practically estimated with groud-truth preference data, focusing on the preference fitting ability of reward models. we call this novel optimization form as ## experiments we verify the effectiveness of apo on the help-ful&harmless (hh) dataset @xcite with alpaca @xcite and llama-2 @xcite as the base llm. due to the limitation of computational resources, we find the original ppo @xcite has very low training efficiency, especially during the online sampling process. since recent offline alignment methods have shown competitive performance to ppo @xcite , we choose rjs (dong et al., 2023), rrhf @xcite , and dpo @xcite as baselines instead. ## conclusion we proposed an adversarial preference optimization (apo) framework to enhance the llm alignment. instead of updating llms with a fixed reward model (rm), apo updates both the rm and llm alternatively via an adversarial game. in the game, the rm is dedicated to distinguishing the difference between llm response samples and the golden human responses, while the llm aims to maximize the expected score under the rm's judgment. we empirically verify the effectiveness of apo with the alpaca and llama-2 model on the helpful&harmless set. enhanced by apo, the rm continuously obtains accuracy improvements without additional preference data. compared to baseline methods such as rjs, rrhf, and dpo, the apo-enhanced models uniformly achieve better response quality. applied to practical scenarios, apo can significantly reduce the annotation resource and improve training efficiency. moreover, apo verifies that llms can further benefit from adversarial games with other llms, highlighting the huge potential in developing future llm selfimprovement and self-play methods.