Semi-Supervised Reward Modeling via Iterative Self-Training (2024)

Abstract: Reward models (RM) capture the values and preferences of humans and play a central role in Reinforcement Learning with Human Feedback (RLHF) to align pretrained large language models (LLMs). Traditionally, training these models relies on extensive human-annotated preference data, which poses significant challenges in terms of scalability and cost. To overcome these limitations, we propose Semi-Supervised Reward Modeling (SSRM), an approach that enhances RM training using unlabeled data. Given an unlabeled dataset, SSRM involves three key iterative steps: pseudo-labeling unlabeled examples, selecting high-confidence examples through a confidence threshold, and supervised finetuning on the refined dataset. Across extensive experiments on various model configurations, we demonstrate that SSRM significantly improves reward models without incurring additional labeling costs. Notably, SSRM can achieve performance comparable to models trained entirely on labeled data of equivalent volumes. Overall, SSRM substantially reduces the dependency on large volumes of human-annotated data, thereby decreasing the overall cost and time involved in training effective reward models.

## introduction reinforcement learning from human feedback (rlhf) @xcite @xcite ) is central to the advancement of large language models (llms), including gpt-4 @xcite and claude @xcite . in rlhf, a reward model is trained to capture common human values and preferences by learning from pairwise preference data. subsequently, language models are refined based on the reward signal to ensure that their actions are aligned with human preferences. this enables language models to perform complex tasks with intricate and challenging objectives, including mathematical reasoning @xcite , code generation @xcite , summarization @xcite , among others. while the resulting aligned language models produced by rlhf receive considerable attention, the importance of reward models is often overlooked. the accuracy of reward models in capturing human preferences is crucial for the effectiveness of rlhf. moreover, beyond the application in rlhf, the ability to annotate preferences for prompt-response data enables reward models to serve as a valuable tool for a broader range of alignment approaches, including rejection sampling finetuning (rsf) @xcite @xcite and direct preference optimization (dpo) @xcite . however, training reward models necessitates a significant volume of human-annotated preference data. furthermore, in real-world applications, it is typical to encounter examples that deviate from the predefined training domains, and it is impractical to gather annotated preference data for every conceivable domain, presenting a significant barrier to the deployment of reward models. this challenge underscores the need to explore methods for enhancing reward models without relying extensively on large datasets of human-annotated preferences. to enhance the efficiency of data utilization in reward model training, we propose semi-supervised reward modeling (ssrm), which efficiently utilizes unlabelled data to improve reward models. our methodology stems from seminal works in the field of boosting @xcite and semi-supervised learning @xcite , which aims at converting weak models to strong models with minimal requirement of labeled data. given a pre- base model 𝜋 ! "#$ 𝐷 % srm model 𝜋 ! & 𝐷 ' 𝐷 ( (i) pseudo-label ssrm model 𝜋 ! ( (iii) srm on augmented data t srm 𝐷 % 𝑃(𝑎 ! ≻ 𝑎 " |𝑥) ! 𝑦 = 1 append (ii) confidence threshold example preference annotation prompt-responses triplets: [context] how do you become a social worker?\n [response a] do you have any previous experience or training? [response b] a social worker helps disadvantaged people in our society. some people become social workers by studying social work at university. another way is to study a related area at university, and then do an apprenticeship in a relevant field. reward model response: b trained language model and an unlabeled dataset with prompt-response pairs, ssrm iteratively executes the following steps: (i) pseudo-labeling: assign pseudo-labels to the unlabeled examples based on their predicted preferences, (ii) confidence thresholding: employ confidence threshold to selectively retain examples where the model exhibits high certainty in its predictions, (iii) supervised reward-modeling (srm): finetune on the filtered subset of data to enhance the reward model. through extensive experiments on various language models with parameter counts ranging from 0.4b to 8b, we demonstrate that ssrm effectively improves the performance of reward models without additional labeling costs. notably, our findings reveal that reward models trained using ssrm exhibit performance closely approaching models trained entirely through traditional supervised methods on equivalent volumes of data. this underscores the efficacy of ssrm in utilizing unlabeled data to mirror the learning gains typically achieved only with labeled datasets. ## semi-supervised reward modeling semi-supervised reward modeling (ssrm) utilizes a small amount of labeled data and a large amount of unlabeled data to efficiently enhance the capabilities of reward models. in section 2.1, we introduce ## experiments our experiments are designed to evaluate the scalability and efficiency of ssrm across a spectrum of model sizes and configurations. we use a confidence threshold of 0.8, and more implementation details can be found in appendix a. ## conclusion in this work, we address the substantial reliance of reward model training on extensive humanannotated preference data. we introduce semi-supervised reward modeling (ssrm), a method that mitigates this issue by effectively utilizing unlabeled data in conjunction with limited labeled data. the ssrm process consists of three primary steps: pseudo-labeling the unlabeled examples, selecting those examples with high prediction confidence, and finetuning the model on this augmented dataset. our extensive experimental evaluations across models of varying sizes demonstrate that ssrm significantly enhances the performance of reward models with minimal requirement of labeled data. furthermore, the performance of models trained using ssrm closely approaches that of models trained with equivalent volumes of fully supervised data. this performance is further supported by detailed analyses of calibration and prediction confidence, which underscore the robustness and effectiveness of ssrm. overall, ssrm offers a highly efficient strategy for improving reward models, significantly reducing the need for costly and time-consuming data annotation.