Aligning Large Language Models via Fine-grained Supervision (2024)

Abstract: Pre-trained large-scale language models (LLMs) excel at producing coherent articles, yet their outputs may be untruthful, toxic, or fail to align with user expectations. Current approaches focus on using reinforcement learning with human feedback (RLHF) to improve model alignment, which works by transforming coarse human preferences of LLM outputs into a feedback signal that guides the model learning process. However, because this approach operates on sequence-level feedback, it lacks the precision to identify the exact parts of the output affecting user preferences. To address this gap, we propose a method to enhance LLM alignment through fine-grained token-level supervision. Specifically, we ask annotators to minimally edit less preferred responses within the standard reward modeling dataset to make them more favorable, ensuring changes are made only where necessary while retaining most of the original content. The refined dataset is used to train a token-level reward model, which is then used for training our fine-grained Proximal Policy Optimization (PPO) model. Our experiment results demonstrate that this approach can improve LLM performance by up to 5.1% in terms of win rate against the reference model, compared with the traditional PPO model.

## introduction one key objective in advancing large language models (llms) is to ensure safe, beneficial human interaction. however, current pre-trained models, mostly trained on web and book texts, often generate biased or toxic text, misaligning with human intentions. to address this issue, numerous studies @xcite @xcite @xcite have integrated human feedback into the training process. a significant advancement is reinforcement learning from human feedback (rlhf) @xcite , which usually consists of two phases: first, a reward model (rm) is trained from preference data, which comprises various responses alongside their human-assigned preference scores for a given prompt. then, this reward model is applied to optimize a final model using proximal policy optimization (ppo) @xcite . recent works @xcite @xcite @xcite discovered limitations of the current rm, specifically their misalignment with human values. this misalignment stems from two main issues: (i) the presence of incorrect and ambiguous preference pairs in the humanlabeled datasets; (ii) the limited insight inherent in sequence-level feedback. specifically, from a data collection standpoint, the task of comparing the overall quality of model outputs is challenging for human annotators when outputs exhibit both desired and undesired behaviors in different parts. moreover from the rm perspective, the reliance on preference-based data labeling leads to sparse training signals. this sparsity discourages the model's ability to distinguish finer details between responses and further limits the capacity for reward optimization. to tackle this challenge, we propose the following two-fold contributions as illustrated in figure 1: • we introduce a new data collection approach that asks annotators to edit responses from existing rm datasets to be more preferable. by comparing the original and edited responses, we obtain detailed token-level insights that are essential for training our fine-tuned reward model. experiment results using alpacafarm @xcite environment indicate that our proposed approach improves llms' performance up to 5.1% against the baseline in terms of win rate, given the same amount of data for training. ## method in this section, we introduce our approach to finegrained data collection through editing and tokenlevel reward modeling. ## limitations although the empirical results show that our approach achieves better performance in model alignment, we struggle to provide rigorous mathematical proof to conclusively demonstrate the effectiveness of this reward allocation strategy, specifically in equation 4. ## conclusion in this paper, we introduce a fine-grained rlhf framework that includes a data collection technique alongside a token-level reward model. this approach enables better value alignment by learning a more accurate reward model, facilitating faster convergence for ppo. our experimental results show performance improvement based on automatic evaluations compared to the baseline method.