Back to Basics: Revisiting REINFORCE -Style Optimization for Learning from Human Feedback in LLM s (2024)

Abstract: AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. Proximal Policy Optimization (PPO) has been installed by the seminal literature as the standard method for the RL part of RLHF. However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit how alignment from human preferences is formulated in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed “RL-free” methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics allows benefiting from online RL optimization at low cost.

## introduction state-of-art large language models (llms) are typically pre-trained on tremendous amounts of text @xcite @xcite @xcite spanning trillions of tokens. these training corpora often contain many complex preferences, relations, and intentions that may not all be desirable for an llm to exhibit. a question of great interest to both the research and wider practitioner community is how to align these models to human preferences? despite being the focus of considerable research effort @xcite @xcite @xcite , there is a lack of consensus regarding the optimal approach to achieve this goal. reinforcement learning from human feedback (rlhf), one of the most widely regarded alignment approaches, directly borrows from traditional rl literature and uses techniques such as proximal policy op-timization (ppo) to maximize the reward score produced by a reward model that is typically trained as a binary classifier on pairs of completions labeled by human annotators. while ppo has become a canonical approach cemented in popularity through its usage in the seminal literature on rlhf @xcite @xcite , getting ppo to work in practice is non-trivial for non-rl specialists and comes with known issues: recent works propose "rl-free" methods such as dpo @xcite , @xcite or iterative fine-tuning approaches to llm preference training @xcite @xcite . however, these works fail to question whether a simpler solution within an rl paradigm exists. instead, all these approaches attempt to answer this question by stripping all rl components from rlhf and the difficulties that come with it @xcite @xcite @xcite @xcite . iterative fine-tuning techniques rely solely on a powerful reward model to identify a subset of samples to train on, while dpo and ipo avoid both reinforcement learning and training a separate reward model by directly learning from human feedback. in contrast to these approaches, we remain in the rl paradigm, but instead return to basics. the core question we seek to explore in this work is can we avoid the computational and optimization complexity of ppo while preserving performance?. we isolate several key differences between traditional deep-rl settings which originally motivated ppo and typical human-preference learning settings for llms. we note that ppo, as an approach, emphasizes stability across iterations, aiming to train an effective policy with the premise of small, stable updates. ppo was designed for a regime where off-policy gradient updates are large enough to introduce instability. this regime dominates traditional deep-rl benchmarks @xcite . however, in this work, we posit that the setting of rlhf, which involves fine-tuning a pre-trained llm, is lacking in these characteristics. in contrast to traditional deep-rl settings, the initialization of the policy, in the form of a pretrained and supervised fine-tuned (sft) model, is far from a random parameterization. while the conceivable search space is enormous, due to the pretraining and sft stages, only a far smaller subset of tokens is likely to be generated as the probability mass is concentrated on these few tokens. thus, while traditional deep-rl settings require strong regularization to reduce the high variance of the gradient estimators; we observe empirically this is less of a practical concern in rlhf and motivate a less computationally expensive method that preserves robustness @xcite . furthermore, we revisit how learning from human preferences is formulated in the context of rl where generating each token is modeled as an action, and each partial sequence, starting with the prompt, is seen as a state. in practice, this modeling assumption for ppo method is often voided. we argue and show that the modeling of partial sequences is unnecessary in this setting where rewards are only attributed to full generations, with no true rewards for any intermediary tokens in the generation. thus, it is more appropriate and efficient to model the entire generation as a single action with the initial state determined by the prompt. given these observations, while keeping simplicity as a guiding principle, we explore the use of the reinforce estimator @xcite and its multi-sample extension reinforce leave-one-out (rloo) @xcite to optimize the sequence-level objective. we break apart ppo and show that the most basic policy gradient algorithm, vanilla policy gradient reinforce consistently outperforms ppo. ppo is unnecessarily complicated for a pre-trained llm environment. unlike ppo, we can use reinforce to directly optimize the full trajectory (sequence) return coupled with unbiased baselines, whereas actor-critic algorithms @xcite , such as ppo, bootstrap off intermediary state value-functions to reduce variance at the cost of introducing bias into the estimator. we arrive at consistent results across models including llama @xcite , pythia @xcite and datasets such as the anthropic helpful & harmless @xcite and tl;dr summarize @xcite : ## background the original rlhf pipeline for llms proposed in (ziegler et al., 2020) consists of three stages: while reinforcement learning approaches share the components above, techniques differ in the formulation of the reward. to understand these differences, we introduce ppo and distinct alternatives such as reinforce and reinforce leave-one-out in the following sections. ## conclusion at a high level, this work posits that the rlhf setting of fine-tuning an llm has a strong initialization of the policy, which coupled with further conditioning on a prompt, alleviates historical concerns with high variance and large action spaces. we support this position with empirical results, showing that techniques such as reinforce & rloo rarely used in traditional rl settings due to high variance outperform ppo. we also show the improved robustness of rloo relative to iterative fine-tuning methods like raft -achieving a best of both worlds sweet spot.