GDPO : Learning to Directly Align Language Models with Diversity Using GF low N ets (2024)

Abstract: A critical component of the current generation of language models is preference alignment, which aims to precisely control the model’s behavior to meet human needs and values. The most notable among such methods is Reinforcement Learning with Human Feedback (RLHF) and its offline variant Direct Preference Optimization (DPO), both of which seek to maximize a reward model based on human preferences. In particular, DPO derives reward signals directly from the offline preference data, but in doing so overfits the reward signals and generates suboptimal responses that may contain human biases in the dataset. In this work, we propose a practical application of a diversity-seeking RL algorithm called GFlowNet-DPO (GDPO) in an offline preference alignment setting to curtail such challenges. Empirical results show GDPO can generate far more diverse responses than the baseline methods that are still relatively aligned with human values in dialog generation and summarization tasks.

## introduction the goal of language model (lm) alignment is to steer the model's generation to produce outputs deemed desirable to human needs and values. reinforcement learning with human feedback (rlhf) is one such critical technique, as evidenced by notable applications such as chatgpt @xcite and claude @xcite . the classical rlhf pipeline involves training the reward model from human feedback and optimizing the policy with the learned reward model by rl, e.g. proximal policy optimization (ppo) @xcite . despite its effectiveness, this pipeline is known to be sample-inefficient and unstable. moreover, its optimal performance hinges on the code-level details and meticulously tuned hyperparameters, making it difficult to reproduce its success with limited computational resources. to simplify this complex rlhf pipeline, recent works have explored offline learning algorithms such as direct preference optimization (dpo) @xcite , which aims to improve the efficiency and stability of rlhf by leveraging human feedback data to derive reward signals directly. while convenient and compute-efficient due to the offline nature of its training, theoretical results suggest that dpo tends to overfit on the reward signal @xcite and learns to reject undesired responses at a faster rate than it learns to accept desired responses, limiting the model's learning capacity @xcite . to overcome these challenges, other works @xcite @xcite have proposed regularized objectives, but none directly aims to model the diversity of the distribution. instead, they tend to settle around local modes in reward distributions, which may be suboptimal. this lack of diversity may hinder its applicability to creative usecases @xcite or under-represent certain demographics in the lm's responses @xcite . in this work, we directly tackle the goal of preference alignment from the perspective of bayesian inference. in particular, we utilize gflownets @xcite , which has recently been introduced as a principled method for amortized sampling of multimodal distributions in proportion to a given reward distribution. sampling proportionally to the reward distribution results in diverse yet high-reward samples. while there has been an application of gflownets for tuning llms to induce a latent chain-of-thought @xcite , there is no established method for using gflownets in the context of offline alignment of lms without relying on an explicit reward model. to this end, we propose gflownet-direct preference optimization (gdpo), providing an efficient offline method for language model alignment. similar to dpo, gdpo learns the policy by extracting reward signals directly from the offline preference dataset, but this task is modeled as an inference task using the gflownet. empirically, we show that gdpo can generate more diverse responses than the baselines in both dialogue generation and summarization tasks while remaining aligned with the preference dataset. ## preliminaries we define the token-wise markov decision process (mdp) as a tuple ⟨s, a, f, r, ρ 0 ⟩, where the state space s consists of tokens generated so far, action space a is the vocabulary of tokens, transition f is the string concatenation, and the initial distribution ρ 0 is the distribution over the prompt x. the episode ends when the model generates the end-of-sequence (eos) token (denoted ⊤), from which no future reward is given. the resulting trajectory after iterative sampling from the policy is the response @xmath0:= y 1:n ⊤. for notational simplicity, we shall denote the initial state s 0 := x and the terminal state s f := x; y. ## method as aforementioned in section 2.2.2, equation 3 can be hard to deal with due to intractable posterior terms. in this section, we present gdpo, which is able to overcome these challenges via gflownets and provide an efficient offline method for aligning llms. detailed balance. gflownets are optimized via objectives based on balance conditions. the balance conditions are imposed on the flow network to ensure that the flow is consistent with the underlying dynamics of the graph 1 . for the application in language modeling, we consider the detailed balance (db) @xcite condition, which simplifies the objective and parameterizations. to see this, we note that gflownets can be drastically simplified in the language modeling setting @xcite . since all states in token mdps are terminable (since eos token probability is non-zero at most states), we can parameterize the db condition @xcite dictates that the transition flows must coincide, similar to 1 see appendix b for further details on the different objectives the db condition in markov chains. this follows immediately from the definition of the forward and backward policies, and the detailed balance condition can be written as f (s)π(s ′ | s) = f (s ′ )π b (s | s ′ ) . the original db objective in @xcite is in the following form: following the lm formulation, the db objective can be written in terms of reward and the forward policy foot_0 . letting π b (•) = 1 and f (s) = r(s)/π(s f | s), we have reward model. we define the token-wise reference log reward for the k-th token for each response in the pair as equation 6 . this ensures the model does not deviate too far from the reference model and learns to terminate at appropriate positions. we temper the terminating log probability with hyperparameter γ ∈ (0, 1] to control the strength of the reward signal. we found it helpful to set γ ≤ 0.5. without the terminating log probability reward, the generation may end abruptly because the flow function has been parameterized with the reward by assuming every state is terminable. given pairwise preference data ⟨x, y, y ′ ⟩, we apply terminating flows to the preferred responses by setting p(y ≻ y ′ | x) = 1 y≻y ′ , similar to the assumption made in the dpo objective. we expect that a gflownet-tuned language model (lm) policy will learn to assign credit to each token in a manner akin to dpo, as suggested by concurrent works @xcite . with α ∈ (0, 1], we define the total reward at the k-th token of a response y with the other response in the pair y ′ as: for a full summary of the method, refer to the pseudocode in algorithm 1 foot_1 . ## experiments model and baselines. to investigate the scalability of the algorithm, we fully fine-tune and evaluate a series of opt models, namely opt -1.3b, and -2.7b @xcite . we compare the proposed method against the following baselines: supervised finetuning (sft), ppo @xcite @xcite , dpo @xcite , identity preference optimization (ipo) @xcite , contrastive preference optimization (cpo) @xcite , sequence likelihood calibration (slic) @xcite , and orpo @xcite . the training hyperparameters and implementation details are provided in appendix a. ## results according to figure 1 and 2, gdpo generates significantly more diverse outputs compared to other methods, demonstrating a clear advantage in encouraging creativity and variability in model outputs. additionally, the method scales effectively to larger models, showing consistent performance improvements as model size increases. however, this increase in diversity is accompanied by a higher standard error in win rate. depending on the task, gdpo performs on par with or, in some cases, worse than baseline methods on average. in particular, gdpo struggles with summarization tasks (figure 2 ), likely due to the nature of these tasks, where consistency is prioritized over diversity in outputs. on the other hand, gdpo can still perform on par or slightly worse than the baselines in the dialogue generation task (figure 1 ), with a range stretching to that of high-performing methods such as dpo and ipo. ## conclusion we propose gdpo, a novel approach to language model alignment that leverages the strengths of gflownets to overcome the limitations of traditional rlhf and dpo methods. gdpo simplifies the alignment process by utilizing an offline preference dataset and modeling the task as a bayesian inference problem. our empirical results demonstrate that gdpo not only maintains alignment with human preferences but also generates more diverse responses compared to existing methods. this increased diversity is crucial for applications requiring creative outputs and for ensuring that a broader range of user demographics is adequately represented in the model's responses.