Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding (2024)

Abstract: Recent strides in large language models (LLMs) have yielded remarkable performance, leveraging reinforcement learning from human feedback (RLHF) to significantly enhance generation and alignment capabilities. However, RLHF encounters numerous challenges, including the objective mismatch issue, leading to suboptimal performance in Natural Language Understanding (NLU) tasks.To address this limitation, we propose a novel Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) to amplify the performance of LLMs in NLU tasks. By incorporating label-sensitive pairs into reinforcement learning, our method aims to adeptly capture nuanced label-sensitive semantic features during RL, thereby enhancing natural language understanding.Experiments conducted on five diverse foundation models across eight tasks showcase promising results. In comparison to Supervised Fine-tuning models (SFT), RLLR demonstrates an average performance improvement of 1.54%. Compared with RLHF models, the improvement averages at 0.69%. These results reveal the effectiveness of our method for LLMs in NLU tasks.

## introduction large language models (llms) @xcite @xcite have undergone impressive advancements that transform nlp tasks into a unified text-totext paradigm, achieving robust alignment and generation capabilities through reinforcement learning from human feedback (rlhf) @xcite . particularly, models are required to predict the correct labels in natural language understanding (nlu) tasks, distinct from natural language generation (nlg) tasks. numerous studies have employed "rationales" to assist llms with chain-of-thought (cot) prompting during supervised fine-tuning (sft) stage @xcite . rationale refers to the relevant parts or information that provide explanations or support for the predictions or decisions made by a model. however, @xcite detail a fundamental challenge in rlhf learning schemes: the objective mismatch issue. this arises when the reward model is influenced by human preference data, introducing biases that conflict with downstream evaluation metrics, especially when applied to nlu tasks. in rlhf, comparison data is initially sampled from the sft model and ranked by a labeler. then the policy model is optimized against the reward model that is trained with these pairs to align with human preference. for nlu tasks, the pairs can be categorized into rationale-sensitive and label-sensitive. as illustrated in figure 1 , we provide an example where three answers sampled from the sft model for the same instruction. if two answers have the same label and different rationales, they form a rationale-sensitive pair, with mr agnews ar mrpc qqp mnli sst-2 tasks 0 20 40 60 80 percentage 93.46% 92.49% 75.43% 81.09% 86.38% 78.62% 94.62% 6.54% 7.51% 24.57% 18.91% 13.62% 21.38% 5.38% rationale-sensitive label-sensitive the more reasonable rationale considered superior. in contrast, if two answers have different labels, they form a label-sensitive pair, with the correct label deemed superior. however, we observed that the pairs sampled from the sft model mainly fall into the category of rationale-sensitive. figure 2 shows the specific distribution ratios of pairs across several nlu tasks. the percentage of rationalesensitive pairs exceeds 75%, and in datasets like sst-2, mr, and agnews, surpasses 90%. the severe imbalance in the distribution of pairs leads the model to prioritize the quality of rationales over the correctness of labels during rlhf training, which conflicts with the evaluation metric (mostly label accuracy) of nlu tasks. a detailed analysis is presented in section 4.2. to address this challenge, our paper proposes a reinforcement learning framework enhanced with label-sensitive reward (rllr) for nlu tasks. firstly, we leverage gpt-4 to generate rationales corresponding to the gold labels of the training data. the sft model is trained with rationales, incorporating cot prompting to enhance comprehension abilities. secondly, we generate rationales for the incorrect labels (relative to the gold labels). unlike rlhf, which uses human intervention to rank sentences, rllr automatically constructs label-sensitive pairs for training the reward model based on the correctness of the label. the comparison data is initially sampled from the trained sft model. finally, we train the policy model against the label-sensitive reward model with proximal policy optimization (ppo) to prioritize the correctness of labels. furthermore, optimizing with mixed rewards from the label-sensitive and rationale-sensitive reward models, rllr mixed ensures both the accuracy of labels and the quality of rationales. extensive experiments on eight nlu tasks demonstrate that our method consistently outperforms the sft baseline by an average of 1.54% and the rlhf baseline by an average of 0.69%, while also exhibiting higher quality in rationales generation. our contributions are summarized as: ## related work reinforcement learning from human feedback. llms have demonstrated commendable performance, leveraging rlhf to achieve notable alignment and generation capabilities @xcite @xcite . rlhf aims to optimize the policy language model to generate content that is desired by humans. recently, some research endeavors have uncovered inherent challenges in rlhf @xcite , including feedback type limitations, evaluation difficulties, oversight challenges, etc. several methods have been proposed to mitigate these challenges. @xcite introduce rl from ai feedback (rlaif), training an ai assistant through self-improvement while adhering to constitutional principles that constrain model-generated content. @xcite introduce a fine-grained rlhf framework that uses fine-grained human feedback, such as identifying false sentences or irrelevant subsentences, as an explicit training signal. @xcite introduced a new parameterization of the reward model in rlhf that enables extraction of the corresponding optimal policy in closed form. this allows us to solve the standard rlhf problem with only a simple classification loss. song et al. please give a rationale for the answer "world politics" in a confident tone (regardless of the true answer): ## proposed method in this section, we introduce the training pipeline of our method as illustrated in figure 3 , including supervised fine-tuning, reward model training, and reinforcement learning enhanced with mixed rewards. ## conclusion in this paper, we introduce a reinforcement learning framework enhanced with label-sensitive reward to amplify the performance of llms for nlu. by training the reward model on label-sensitive pairs, which are constructed by generating rationales for the incorrect labels, we mitigate the objective mismatch issue in rlhf, leading to improved performance in nlu tasks. extensive results on 5 foundation models and 8 nlu tasks demonstrate that rllr consistently surpasses the sft baseline by a margin of 1.54%, and the rlhf baseline by 0.69%. by additionally incorporating the label-sensitive and rationale-sensitive rewards, our enhanced rllr mixed method not only maintains the label accuracy comparable to rllr but also achieves rationale quality on par with rlhf. we present an in-depth analysis of rllr, examining the utilization of rationales, reward modeling objectives, and incorporation of multiple rewards during the rl stage. the results and analysis substantiate the effectiveness of our methods in the nlu tasks.