Removing Prompt-template Bias in Reinforcement Learning from Human Feedback (2025)

Abstract: Reinforcement Learning from Human Feedback (RLHF) has become an essential technique for enhancing pre-trained large language models (LLMs) to generate responses that align with human preferences and societal values. Although RLHF has shown promise, the training of reward models (RMs) still faces the challenge of reward hacking , motivating recent works to prevent RMs from finding shortcuts that bypass the intended optimization objectives by identifying simplistic patterns such as response length. Besides the issue of length bias , our work firstly reveals that prompt-template bias learned by RMs can also cause reward hacking when dealing with some marginal samples, resulting in LLMs preferring to generate responses in a specific format after RLHF fine-tuning, regardless of the format requested in the prompt. To this end, we propose a low-cost but effective method, namely Prompt Bias Calibration (PBC), to estimate the prompt-template bias term during reward modeling, which can be utilized to calibrate reward scores in the following RL fine-tuning process. Then, we show that our PBC method can be flexibly combined with existing algorithms of removing length bias , leading to a further improvement in the aspect of enhancing the quality of generated responses.

## introduction reinforcement learning from human feedback (rlhf) has become a critical technique to enable pre-trained large language models (llms) to follow human instructions, understand human intent, and also generate responses that align with human preferences and societal values @xcite @xcite . specifically, rlhf usually trains a reward model (rm) to act as the proxy of * equal contribution † corresponding author human preferences, and then utilizes online reinforcement learning (rl) algorithms to fine-tune the language models to generate responses that can achieve higher expectation rewards, leading to the success of chatgpt and also many other ai applications @xcite . although the paradigm of rlhf has simplified the human data collection procedure, as acquiring human ratings is much easier than collecting demonstrations for supervised fine-tuning (sft), it still requires a large amount of human-annotated preference pairs to train well-performing rms in practice, motivating recent researchers to look for novel alignment methods to bypass rm training @xcite @xcite . however, the original rlhf pipeline is still the primary choice for the industry, because well-trained rms can provide a certain level of generalizability @xcite . in addition to the high costs associated with gathering large amounts of human-annotated preference data, another significant concern often raised about rlhf is the issue of reward hacking @xcite , where the over-optimized rms tend to find some shortcuts to bypass their intended optimization objective, through identifying some simple patterns to distinguish between good and bad responses @xcite . the most widely studied pattern in reward hacking could be the sentence (response) length, and these trained rms can utilize the preference among human raters for longer responses to achieve reward hacking, despite the actual quality of response does not improve with the increase of response length @xcite . thus, to mitigate reward hacking, recent work has primarily focused on estimating the length bias term in the reward scoring process, so that it can be removed in the subsequent rl fine-tuning procedure to further improve the quality of generated responses @xcite . besides the issue of length bias, in the practice of applying rlhf to industrial products, we have observed that the original implementation of rlhf tends to make llms prefer generating responses in a specific format. this observation motivates us to investigate the underlying causes and find a cost-effective solution to address this issue. the main contributions are summarized as follows: ## related works the prevalence of length bias in rlhf has been widely criticized as indicative of reward hacking @xcite , and numerous recent studies have delved into strategies aimed at mitigating the tendency for length increase during the fine-tuning process of rlhf @xcite @xcite . typically, shen et al. @xcite innovatively apply the productof-experts (poe) technique to separate reward modeling from the influence of sequence length, which adopts a smaller reward model to learn the biases in the reward and a larger reward model to learn the true reward. utilizing similar disentangling ideas, chen et al. @xcite jointly train two linear heads on shared feature representations to predict the rewards, one trained to correlate with length, and the other trained to focus more on the actual content quality. ryan et al. @xcite firstly study the length problem in the dpo setting, showing significant exploitation in dpo and linking it to outof-distribution bootstrapping. as for the prompt bias issue, although it has been criticized in the scenario of reward model ensembles @xcite , no studies have yet attempted to analyze its cause and influence on rlhf. we emphasize that our work is the first to fill this gap by proposing a low-cost yet effective method to mitigate the reward hacking induced by prompt-template bias. ## conclusion in this paper, we demonstrate that prompt-template bias in rms can lead to llms, which, after rl fine-tuning, generate responses exclusively in a specific format, irrespective of the variations in the prompt request. thus, we propose a low-cost but effective pbc method, to estimate the prompttemplate bias term during reward modeling, which can be utilized to calibrate reward scores in the following rl fine-tuning process. then, we show that our pbc method can be flexibly combined with existing algorithms of removing length bias, leading to a further improvement in the aspect of enhancing the quality of generated responses.