Prototypical Reward Network for Data-Efficient RLHF (2024)

Abstract: The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to scalability issues for LLMs and complex tasks. Our proposed framework Proto-RM leverages prototypical networks to enhance reward models under limited human feedback. By enabling stable and reliable structural learning from fewer samples, Proto-RM significantly enhances LLMs' adaptability and accuracy in interpreting human preferences. Extensive experiments on various datasets demonstrate that Proto-RM significantly improves the performance of reward models and LLMs in human feedback tasks, achieving comparable and usually better results than traditional methods, while requiring significantly less data in data-limited scenarios. This research offers a promising direction for enhancing the efficiency of reward models and optimizing the fine-tuning of language models under restricted feedback conditions.

## introduction reinforcement learning from human feedback (rlhf) effectively integrates meticulous human judgment with the model's capacity for large-scale data processing @xcite @xcite , enhancing language models' adaptability to human communication styles and preferences @xcite . by utilizing reinforcement learning (rl) over supervised fine-tuning, rlhf captures the complexity of human language, involving emotions, context, and subtle linguistic differences @xcite thereby offering enhanced adaptability and flexibility in human interactions. the success of rlhf hinges on the quality of the reward model @xcite @xcite * corresponding authors @xcite , which guides the rlhf learning process, ensures its accuracy and efficiency @xcite , while preventing deviations from desired outcomes @xcite . a deficient reward model, however, may learn a complex yet inaccurate error surface, leading the model to favor high-scoring yet erroneous solutions @xcite @xcite . this overfitting can result in responses that maximize rewards but diverge from the actual objectives and human preferences @xcite . notably, effective rlhf typically requires extensive data @xcite . this work. to address these challenges, we integrate prototypical networks-instance-based algorithms that learn representative prototypes for similar examples to facilitate tasks like classifications or regression @xcite -with the reward model for rlhf. prototypical networks are particularly suitable for few-shot learning as they efficiently extract key features from limited samples for decision-making @xcite . by optimizing the embedding process in the reward model using prototypical networks, we enable the reward model to learn stable and reliable data representation structures with limited sample sizes. this method is especially suitable in enhancing the model's learning and generalization from human feedback samples, given the limitations of sample quantity and the complexity of human preferences @xcite . to enhance the effectiveness of the reward model with limited human feedback data, we propose the prototypical reward model (proto-rm) that decreases reliance on human feedback without compromising the performance of the reward model. the fundamental principle of the reward model is to assimilate human feedback to evaluate and steer the model outputs to meet human expectations and standards. its crucial capability lies in effectively learning and extracting vital parameter information from limited human feedback, thus guiding the model's behavior. our approach focus on learning a method that performs well in few-shot scenarios and is suitable for learning from human feedback samples while preserving the reward model's network structure and enhancing the parameterization capabilities. our method consists of three pivotal steps. first, in sample encoding and prototype initialization, we employ a reward model to encode samples. these encodings serve as the basis for initializing prototypes with a strategically selected subset of the encoded samples. subsequently, we analyze the relationship between these initialized prototypes and the encodings of other samples. second, in prototype update and addition, we continuously refine the sample encodings based on their distances to the prototypes. concurrently, we update the reward model's parameters by validating the predictions generated through the refined encodings. this iterative refinement ensures that prototypes accurately mirror the attributes of samples, enhancing the learning efficacy through human feedback samples. finally, reward model fine-tuning employs the refined prototypes and encodings to train the reward model. this training aims to accurately evaluate and guide the outputs of the language model, thereby improving the performance of llms during the fine-tuning stage. ## problem formulation the primary objective is to train a reward model capable of training a policy that generates highquality texts, as evaluated by humans, using a constrained set of human-annotated data. input. the input of the reward model is a dataset with n examples, where each example consists of a common post x @xmath0, y -∈ y. these summaries are distinguished by human preferences: y + is the preferred (chosen) response with annotation z + @xmath1, and y -is the less preferred (rejected) response with annotation z - @xmath2, where @xmath3, rejected}. output. the outputs consist of 2 components: ## methodology the primary objective is to train a reward model that predicts which answer {y + i , y - i } is better according to human judgment, given a prompt x i . ## experiments in this section, we first compare the consistency of annotations between proto-rm and baseline reward model (baseline rm) with real human feedback on prompt-answer text pairs. subsequently, we contrast the differences in text quality algorithm 1 reward model with prototypical networks 1: input: @xmath4, y + i , y - i ), (z + i , z - i ) n @xmath5, where each z + , z -∈ @xmath6, rejected} 2: output: the predicting score pair s(x, y + , y -) = (s + , s -) and the reward model f ϕ 3: initialize k prototypes through prototype initialization 4: for minibatch b @xmath7: perform prototype update and addition and estimate λ according to eq. 7 6: for (x i , y + i , y - i ) ∈ b r do 7: converts (x i , y + i , y - i ) into encodings e (x i ,y + i ) and e (x i ,y - i ) 8: for y * @xmath8, y -do 9: allign e (x i ,y * i ) according to eq. 4 10: calculate d i,@xmath9,y * i ) , p k ) for p @xmath10, and d i,@xmath11: update the embedding according to eq. 3 12: if min d i,@xmath12: create the k + 1-th prototype p k+1 using e (x i ,y * i ) ; increment k by 1 14: end if 15: compute s (x i ,y * i ) though annotation prediction 16: end for 17: ## conclusion our research demonstrates the efficacy of prototypical networks in refining rlhf, especially in scenarios with limited human feedback. the enhanced reward model shows a marked improvement in aligning llm outputs with human preferences, as evidenced by our experimental results. future works can explore the application of our method to more diverse and extensive datasets to further validate its effectiveness and adaptability. limitations there are a few limitations to the current framework. first, the proto-rm performs best for open-domain pairwise human preference tasks that focus on alignment with human judgments. it remains unknown if proto-rm yields substantial improvements in scenarios involving @xmath13. second, we focus our study on the englishlanguage human preference data. this source mainly represents the english-language nlp study. our study does not include all the necessary research from other nature languages. we examine how often nlp researchers cite older work by analyzing factors such as the mean age of citations. our findings indicate a link between these factors and the frequency of citations. however, these links do not prove causation. more studies are needed to understand the reasons behind the citations of older papers.