Inverse- Q *: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data (2024)

Abstract: Reinforcement Learning from Human Feedback (RLHF) has proven effective in aligning large language models with human intentions, yet it often relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and present challenges in sample efficiency and stability. In this paper, we introduce Inverse-Q*, an innovative framework that transcends traditional RL methods by optimizing token-level reinforcement learning without the need for additional reward or value models. Inverse-Q* leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model’s responses, facilitating more granular and flexible policy shaping. Our approach reduces reliance on human annotation and external supervision, making it especially suitable for low-resource settings. We present extensive experimental results demonstrating that Inverse-Q* not only matches but potentially exceeds the effectiveness of PPO in terms of convergence speed and the alignment of model responses with human preferences. Our findings suggest that Inverse-Q* offers a practical and robust alternative to conventional RLHF approaches, paving the way for more efficient and adaptable model training approaches.

## introduction reinforcement learning from human feedback @xcite is a mainstream approach for aligning large models to human intentions, demonstrated in applications such as chat-gpt @xcite and llama3 @xcite . the rlhf framework involves modeling a reward function from preference data and learning an optimal policy through ppo @xcite , which also estimates expected returns, translating as an efficient alternative to ppo, direct preference optimization @xcite aligns large models from the perspective of contextual bandits, not token-level decisions @xcite . dpo optimizes preference reward loss directly through reward model loss, affecting the probability margins of preference pairs. similar methods like rso @xcite , rest @xcite , and rest-em @xcite train policies to fit optimal prior distributions on predefined response sets, avoiding the need for a critic model. however, these methods still require additional supervisory signals, such as a reward model, to enhance response quality, leading to trade-offs in labeling costs and accuracy. while direct optimization methods generally overlook token-level preference modeling, some efforts @xcite have explored using reward assignment to refine feedback signals, though these enhancements mainly bolster method stability rather than guide updates. a crucial observation is that direct optimization methods still require the logits of entire response sequences to construct the loss function due to the need for differentiability in back-propagation. lacking corresponding advantage function modeling, such constructions cannot naturally generalize to token-level process supervision. based on this insight, we hypothesize: is there a special trajectory estimation whose feedback signal can naturally generalize to dense reward function modeling within the token mdp, thereby automatically constructing advantage function interpretations for each token? similar properties were demonstrated in r2q* @xcite , where dpo training could implicitly learn the optimal q-function and mimic controllable decoding, but it required pre-labeled preference data for obtaining the reference distribution under the optimal policy. in this work, we estimate the conditionally optimal distribution for current inputs on single dialogue data without additional labeling or external supervision. we introduce inverse-q*, an algorithm that optimizes the same objective as ppo (maximizing the advantage function) with enhanced flexibility and easier implementation. our method, an inverse problem of dpo training, assigns token-level reward feedback via an estimated policy, optimizing the large model online within the mdp framework. overall, inverse-q* exhibits similar sample utilization efficiency and supervision granularity as ppo, providing token-level rl training across all sampling outcomes without relying on additional reward models or value models, thus performing excellently in terms of labeling and computational resource demands. the process of inverse-q* is illustrated in figure 1 , and we have conducted extensive experiments to demonstrate the efficacy of our framework in low-resource rlhf training. inverse-q* has shown the capability to achieve or even exceed the effectiveness of ppo training. our contributions can be summarized as follows: ## preliminaries in this section, we first introduce the classical rlhf framework in llm alignment, followed by a description of how this modeling is tied to direct alignment methods (in the case of dpo). given a prompt x * sampled from the dataset @xmath0, y i )} n @xmath1, policy model provides a multi-token response y * = (y 0 , . . . , y t ) to complete a full interactive dialogue process. to align with the output format of language models throughout this chapter, we use y t-1 = (x * , y 0 , . . . , y t-1 ) to denote the current state at time t in the rl context, where y t represents the policy action at token level. most rlhf algorithms require training a reward function from human preference data to provide online feedback on model outputs. a preference data pair (x, y w , y l ) typically begins with the same initial prompt and receives a corresponding reward score at termination, and the probability of preferring τ w over τ l is given by: this modeling is subsequently used to optimize the generate policy of llms by improving the preferring probability of model responses over older ones. however, human preference annotations typically only exist at the response or sentence level, so the reward model cannot directly provide gradient signals action by action for optimization. ppo artificially defines token-level rewards with entropy bonuses to adhere to the bradley-terry preference modeling @xcite as follows: based on the above definition, ppo aims to maximize the expected reward at each token while ensuring that the learned policy does not diverge significantly from a reference model. for a given input y s , the optimal policy is represented as: on the contrary, dpo utilizes a contextual bandits setting to circumvent token-level reward allocation issues. assuming an implicit reward model r that scores all potential responses {y * i } m @xmath2, the closed-form solution of the policy model under a kl-constrained contextual bandit optimization problem can be expressed as: this modeling is subsequently used to compute standard reward model losses for updating policy distributions. ## conclusion in this article, we propose the inverse-q* algorithm, which has demonstrated comparable sample utilization efficiency and supervision granularity to ppo, achieving token-level reinforcement learning across all sampling outcomes without the need for additional reward or value models. this efficiency significantly eases the demands on labeling and computational resources. extensive experiments validate the effectiveness of the inverse-q* framework in low-resource rlhf training, showing its potential to match or even surpass the performance of ppo training. our method has proven to significantly enhance the alignment of large language model responses with human preferences, achieving faster convergence compared to traditional rlhf methods such as ppo and dpo.