Countering Reward Over-Optimization in LLM with Demonstration-Guided Reinforcement Learning (2024)

Abstract: While reinforcement learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations’ and LLM’s rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation.We show the effectiveness of RCfD in three RL language tasks, where it achieves comparable performance to carefully tuned baselines while mitigating ROO.

## introduction reinforcement learning (rl) has long been used to train conversational agents, ranging from designing dialogue strategies @xcite to language modelling @xcite . while supervised learning excels at pre-training llms @xcite , rl stands out for finetuning llms. it allows optimizing non-differentiable objectives @xcite , improving sequence-planning in goal-oriented dialogues @xcite , or aligning llms with human preferences (rlhf) @xcite , which leads to more helpful and harmless llms @xcite . finetuning llms with rl typically involves scoring their utterances with a reward function, which is then maximized using online rl methods. unfortunately, this optimization process is known to be brittle if not carefully controlled @xcite , reducing language diversity @xcite , generating unnatural language patterns to artificially inflate rewards @xcite , or altering the llm semantics and syntax @xcite . this phenomenon has recently been referred to as reward over-optimization (roo) 1 . roo may be mitigated by incorporating a klregularization term to anchor the finetuned model to its initial human-like language policy @xcite . however, calibrating the kl term requires careful hyperparameter tuning, which is computationally expensive when finetuning llm with online rl @xcite . offline approaches, such as direct preference optimization (dpo) @xcite , attempt to address roo by bypassing the reward estimation and directly maximizing user preferences through pairwise comparisons. unfortunately, these methods also suffer from roo, albeit not optimizing the reward explicitly, and still requires careful kl regularization @xcite . in other words, these attempts to address roo primarily focus on constraining the language policy, leaving the reward objective itself unaddressed. besides, it cannot be generalized beyond pairwise data, limiting it to rlhf settings only. this paper proposes a novel approach, reward calibration from demonstration (rcfd), to tackle roo in llms. inspired by demonstration-guided rl @xcite , rcfd utilizes human demonstrations and a reward model to guide the llm towards generating outputs that achieve similar rewards to those of the demonstrations. this shift from directly maximizing the reward function to calibrating it based on demonstrations helps prevent llms from exploiting the reward model and encourages more natural language generation. furthermore, unlike pure imitation learning, rcfd operates at the sequence level, mitigating exposure bias @xcite and promoting greater diversity in the generated text. we conducted a series of experiments to investigate the effectiveness of rcfd. first, we apply rcfd to maximize the language model's sequence log-likelihood with rl. this experiment demonstrates that rcfd prevents the language degeneration typically observed in rl while avoiding the compounding errors associated with imitation learning. next, we optimize rcfd objectives on two rl language tasks, achieving performance comparable to tuned baselines. this showcases rcfd's ability to effectively address roo while maintaining task performance. finally, we explore rcfd in multi-reward settings, where the goal is to optimize multiple, potentially conflicting rewards. by targeting a point on the pareto frontier through demonstrations, rcfd controls the optimization process. our experiments provide strong evidence that recalibrating the reward objective with demonstrations mitigates roo and offers a promising approach for tackling complex language rl tasks where human demonstrations are available. reward over-optimization (roo) rl was successfully used in multiple language tasks ranging from language modelling (de masson d' @xcite , translation @xcite , summarization @xcite , code generation @xcite , instruction following @xcite or question answering @xcite . however, rl methods were quickly reported to exploit language metrics @xcite , either creating emergent language @xcite or overfitting text classifiers @xcite and user preference models @xcite . more generally, whenever maximizing the reward function over a certain point starts lower the ground truth performance, this can be referred to as reward overoptimization (roo) @xcite . roo has two main origins: (i) the absence of grounding: as solely trained on optimizing scores, llms can become detached from human language @xcite , (ii) the optimization of imperfect reward models @xcite . countering roo reward over-optimization is often mitigated by tying the finetuned model to its base distribution through a kl regularization @xcite @xcite . while simple, this method has multiple variants, e.g., using the kl in the loss @xcite or in the reward @xcite , using decay heuristics @xcite or altering the referent distributions @xcite . however, kl-tuning requires costly crossvalidation as it is impossible to predict the final impact of the kl regularization before training @xcite . as rcfd targets the reward distributions from demonstrations, the resulting llm behavior is far more predictable, making it an apriori regularization method as explored in 5.3. another strategy implemented by dpo @xcite avoids modeling the reward function by leveraging pairwise comparisons. while this circumvents reward imperfection issues, dpo remains susceptible to overfitting @xcite , with kl-regularization only marginally regularizing the training @xcite . more generally, dpo is designed explicitly for optimizing rewards derived from preference models, mak-2 12448 ing it inoperable in rl language tasks where llm completions are scored individually, e.g., success scores @xcite or classifiers @xcite . in particular, we explore in section 5.1 and 5.2 two settings where dpo cannot applied, demonstrating the interest of rcfd beyond the restricted use cases of rlhf. closer to our work, @xcite identify proxy points where roo occurs and retrain the llm by dynamically reweighting the rewards not to exceed the proxy points, avoiding the roo regime. while @xcite and rcfd both recalibrate the reward, rcfd avoid computing the proxy points using demonstrations, requiring less compute and no gold-standard metrics. ## reward calibration from demonstration dgrl for llm: dgrl combines a demonstration dataset in addition to the reward function in rl. formally, the dataset @xmath0, y n d ) n @xmath1. given the rm and a prompt x, we note r rm (x, y) the reward of the llm's completion, and r rm (x, y d ) the reward of the demonstration. ## experimental setting we first use the log-likelihood optimization problem in llms to closely examine the issues that arise with standard rl and imitation learning. this helps us better understand the motivation behind rcfd. then, we evaluate rcfd in a single reward setting. we confirm that rcfd performs similarly to existing best baselines while mitigating roo. finally, we show that rcfd successfully handles multi-reward objectives by using demonstrations to guide llms toward the desired behavior. ## results in this section, we derive the results of the three use cases: sequence-level log-likelihood calibration problem, single-reward optimization, and multireward optimization. sequence log-likelihood lessening with sft in figure 2 , the average log-likelihood of sentences generated by the initial llm diminishes with longer sentences (green line). on the contrary, when we evaluate the log-likelihood of the demonstrations with the initial model, we do not observe this log-likelihood loss along the generation. this shows the existence of the exposure bias. importantly, this exposure bias is barely reduced when performing sft with the demonstraif carefully tuned, optimizing r β can match the reward distribution, but subtle changes in β also induce drastic behavior changes. when β = 0, the lm achieves near-optimal rewards, yet the policy is degraded (naturalness close to 0), illustrating an instance of roo. ## conclusion this paper introduces rcfd, a novel rl objective leveraging demonstrations to guide finetuning in llms and mitigating roo. instead of complex parameter tuning, rcfd calibrates the reward distribution by aligning it with the reward distribution of the demonstrations. hence, rcfd shifts the focus of rl training from tuning parameters to collecting demonstrations, leading to highly predictable model behavior, a valuable asset when dealing with large models or intricate reward structures. finally, compared to classic sft methods, rcfd demonstrates superior utilization of demonstrations when a reward model is available. beyond its practical applications, rcfd also opens doors to a less explored perspective on im-8 itation learning. we suggest that targeting human reward distributions could be a promising proxy for imitating human behavior without accessing the full demonstrations, potentially exceeding stepby-step imitation approaches like sft. further exploration of this avenue is left for future work.