The Accuracy Paradox in RLHF : When Better Reward Models Don’t Yield Better Language Models (2024)

Abstract: Reinforcement Learning from Human Feedback significantly enhances Natural Language Processing by aligning language models with human expectations. A critical factor in this alignment is the strength of reward models used during training. This study explores whether stronger reward models invariably lead to better language models. In this paper, through experiments on relevance, factuality, and completeness tasks using the QA-FEEDBACK dataset and reward models based on Longformer, we uncover a surprising paradox: language models trained with moderately accurate reward models outperform those guided by highly accurate ones. This challenges the widely held belief that stronger reward models always lead to better language models, and opens up new avenues for future research into the key factors driving model performance and how to choose the most suitable reward models.

## introduction language models (lms) have made remarkable progress, achieving close-to-human capabilities in a wide range of tasks @xcite @xcite @xcite . while traditional fine-tuning has been effective, it often suffers from exposure bias, where models are trained on ground truth data rather than their own predictions, leading to inconsistencies during generation @xcite . additionally, fine-tuning lacks the ability to optimize for sequence-level rewards, limiting its effectiveness in capturing complex, human-like preferences @xcite . rlhf addresses these limitations by incorporating feedback from humans, allowing models to generate more contextually relevant and aligned outputs @xcite @xcite . * corresponding authors. it is commonly assumed that higher accuracy in reward models enhances language model performance because these models provide precise feedback during training @xcite . this perspective suggests that accurate feedback directly improves the effectiveness of lms, especially in complex tasks like machine translation and question answering @xcite . in this paper, we conducted extensive experiments using the qa-feedback dataset @xcite . reward models based on longformer @xcite were evaluated for their binary classification accuracy in predicting task relevance, factuality, and completeness. to ensure fair evaluation, the performance of lms trained with these reward models was assessed using independent high-accuracy models tailored to each task. surprisingly, our findings reveal a paradox: lms achieve their best performance not with the most accurate reward models, but with those of moderate accuracy @xcite , challenging the prevailing assumption that higher reward model accuracy directly correlates with improved outcomes. this result raises important questions about the relationship between reward model accuracy and language model performance in rlhf, warranting further investigation. the main contributions of this study include: ## motivation and problem settings motivation. findings indicate that the strength of reward models in rlhf does not consistently correlate with improved language model performance, challenging the assumption that stronger reward models always lead to better outcomes @xcite . understanding the dynamic relationship between reward model accuracy and language model performance is essential for optimizing rlhf in complex nlp tasks @xcite . this study posits that there exists an optimal range of reward model accuracy that maximizes language model performance @xcite . therefore, the primary aim of this research is to identify this optimal range and examine its implications for various nlp applications. problem settings. this study investigates the effect of reward model strength on language model performance in rlhf, focusing on tasks that evaluate the factuality, relevance, and completeness of generated text @xcite . specifically, reward model strength is defined by binary classification accuracy on test sets @xcite , and language model performance is measured using high-accuracy, independent reward models. formally, for a language model trained with rlhf, this study analyzes how the reward model's classification accuracy (s rm ) and the number of training steps (τ ) affect language model performance (p lm ) @xcite . this relationship is mathematically represented by: the objective is to determine the optimal conditions that maximize language model performance across various tasks, providing insights for the development of more effective rlhf strategies in nlp @xcite . ## conclusion and future work this study demonstrates that lms trained with moderately accurate reward models in rlhf achieve optimal performance, challenging the conventional belief that higher accuracy is always more beneficial. the results show that moderately accurate reward models offer more task-aligned feedback and foster a balanced, stable training process, promoting better generalization. this research highlights the limitations of relying exclusively on highly accurate reward models, as excessive focus on accuracy may lead to suboptimal outcomes. in future work, it will be crucial to further explore the potential overfitting of reward models, particularly in their ability to generalize to out-of-distribution (ood) tasks. techniques such as regularization, data augmentation, and explicit ood evaluation will be key areas of investigation to enhance the robustness of reward models across diverse scenarios and ensure their effectiveness in guiding lms in broader, more complex nlp tasks.