Don’t Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2024)

Abstract: While Reinforcement Learning from Human Feedback (RLHF) significantly enhances the generation quality of Large Language Models (LLMs), recent studies have raised concerns regarding the complexity and instability associated with the Proximal Policy Optimization (PPO) algorithm, proposing a series of order-based alignment methods as viable alternatives. This paper delves into existing order-based methods, unifying them into one framework and examining their inefficiencies in utilizing reward values. Building upon these findings, we propose a new Value-based Calibration (VCB) method to better align LLMs with human preferences. Experimental results demonstrate that VCB surpasses existing alignment methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings.

## introduction large language model (llm) has demonstrated notable capabilities in various areas including text summarization @xcite and code generation @xcite . despite preliminary cleaning, training datasets of llms still harbor considerable amounts of low-quality and potentially toxic content, adversely affecting llms @xcite . a widely adopted solution involves employing reinforcement learning from human feedback (rlhf) @xcite to align llms with human preferences. specifically, rlhf encompasses three phases: (1) supervised fine-tuning (sft); (2) preference sampling and reward learning; (3) rl optimization using the proximal policy optimization (ppo) algorithm @xcite . while rlhf significantly reduces toxic content and enhances response quality, recent studies @xcite have raised concerns regarding the complexity and instability of the ppo algorithm, prompting the exploration of alternative approaches. rrhf @xcite , slic @xcite , and dpo @xcite are representative methods among these alternatives and all of them are based on an intuitive core idea: given a preference dataset d @xmath0, y w , y l )}, where the response y w is preferred over y l for the same prompt x, these methods calibrate response generation probabilities to be aligned with preference orders using contrastive losses. therefore, we refer to such methods as order-based calibration methods. rrhf, for instance, employs the following contrastive ranking loss @xcite : theoretically, order-based calibration methods enable direct alignment of llms, obviating the need for reward models. however, in practice, the high cost of annotating preference data @xcite constrains the scope of preference datasets. therefore, many recent studies @xcite persist in utilizing reward models to automatically augment preference datasets. specifically, this process starts by employing the instruction sft model to produce a series of candidate responses {y 1 , y 2 , • • • , y n } to a prompt x. subsequently, the reward model evaluates and ranks all candidate responses, establishing a preference order {y @xmath1. ultimately, the derived preference order is used to align llms through order-based calibration methods. however, existing order-based methods are initially designed for avoiding the reward model @xcite . most of them disregard the reward values and solely optimize the relative orders, which oversimplifies the training process and has room for improvement. as illustrated in figure 1 , let's consider three responses y 1 , y 2 and y 3 with rewards of 0.1, 0.85 and 0.9, respectively. obviously, responses y 2 and y 3 are almost equally good, whereas response y 1 is significantly inferior. current order-based methods (e.g., dpo and slic) tend to disregard the absolute reward values, only focusing on the relative orders. this may result in the generation probability of y 2 being inappropriately closer to that of y 1 than to y 3 , leading to a potential misalignment. to theoretically address the above limitation, this paper begins with proving that existing order-based calibration methods can be traced back to a single optimization problem under different entropy settings. then, our further investigation reveals that these order-based methods' inability to utilize reward values stems from their elimination of the partition function during the reparameterization process, which also removes the reward function. finally, diverging from using a reparameterization, we suggest employing a difference method to eliminate the partition function, which could preserve the reward function within the loss function. based on the above findings, this paper proposes a new value-based calibration (vcb) method, enabling the utilization of reward values. as shown in figure 1 , our method transcends mere order-based calibration by ensuring that the relative probability gap between responses is directly proportional to their relative reward gap. consequently, responses with comparable rewards will have similar generation probabilities, effectively overcoming the misalignment problem of solely calibrating according to the order of rewards. it is worth noting that, although our proposed method is not the first one to mention utilizing reward values @xcite , vcb is fully grounded in theoretical deduction and logical reasoning, rather than solely on intuition. our contributions are summarized as follows: ## related work due to the variable quality of training data, unsupervised pre-trained llms might not closely align with human preferences, potentially generating unsafe, toxic, biased, or even criminal responses. a widely adopted solution is to use reinforcement learning from human feedback (rlhf) @xcite to align llm outputs with human preferences. the objective of rlhf can be formulated as an optimization problem, described as follows: although ppo has demonstrated remarkable capabilities in llm alignment, its training process is notably intricate and unstable @xcite . consequently, recent studies have explored direct alignment with preference data, such as rrhf @xcite , slic @xcite , and dpo @xcite . although the specific forms vary, these methods share a core idea: calibrating responses' probability orders with their reward preference orders. for any two responses y i and y j , if r(x, y i ) > r(x, y j ), they hope that π(y i |x) > π(y j |x) holds. therefore, we refer to these methods as order-based calibration methods. rrhf @xcite slic @xcite dpo @xcite ) ## unifying rrhf, slic and dpo although rrhf and slic empirically demonstrate their effectiveness and scalability, they are still purely based on intuition and lack theoretical underpinnings. in contrast, dpo conducts a detailed theoretical analysis, elucidating how the loss is derived from the bradley-terry model @xcite . to deepen understanding of these order-based methods and elucidate their limitations in effectively utilizing reward values, this paper further unifies rrhf, slic, and dpo within a single framework. specifically, all these three order-based calibration methods could be traced back to the following optimization problem: theorem 1 if ψ π (y|x) = -α(x)[log π(y|x) + β(x, y)], α(x) and β(x, y) do not depend on the policy π, and α(x) > 0 for all prompts x, the optimal solution of eq.3 is: z(x) = y e r(x,y) α(x) -β(x,y) represents the partition function. detailed proof is in appendix a.1. because estimating the partition function z(x) is usually expensive @xcite , this optimal solution is difficult to be directly utilized in practice. however, eq.5 establishes an equivalence relationship between the reward model and the optimal policy. it could be rearranged as follows: r(x, y) = α(x) log π opt (y|x) + β(x, y) + log z(x) (6) according to eq.6, we can apply a reparameterization to contrastive reward losses and transform them to existing order-based calibration losses. let's take slic as an example. when α(x) = γ, β(x, y) = 0 and using the margin contrastive loss @xcite as the reward training loss: l @xmath2,yw,y l )∼dp max [0, δr(x, y w ) + r(x, y l )] (7) δ represents the margin. y w and y l are a preference response pair. by applying a reparameterization to l r , specifically by replacing r(x, y) according to eq.6, we can obtain the loss function of slic: actually, rrhf and dpo can also be derived in a similar way. the only difference lies in the adoption of different conditional entropy penalties h π ψ (y |x) and reward losses l r . ## the proposed approach in this section, we aim to: (1) introduce a novel alignment loss via value-based calibration; (2) demonstrate the derivation of the proposed valuebased calibration loss from eq.3; ## conclusion large language models (llms) alignment has been shown to greatly diminish the probability of producing biased or illegal content. this paper delves into current order-based alignment methods, exploring why they fail to make effective use of reward values, and further proposes a novel value-based calibration (vcb) method to better align llms with human preferences. experiments demonstrate that vcb surpasses existing order-based methods in both ai assistant and summarization tasks.