Shallow Preference Signals: Large Language Model Aligns E ven Better with Truncated Data? (2025)

Abstract: Aligning large language models (LLMs) with human preferences remains a key challenge in AI. Preference-based optimization methods, such as Reinforcement Learning with Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on human-annotated datasets to improve alignment. In this work, we identify a crucial property of the existing learning method: the distinguishing signal obtained in preferred responses is often concentrated in the early tokens. We refer to this as \textit{shallow preference signals} .To explore this property, we systematically truncate preference datasets at various points and train both reward models and DPO models on the truncated data. \textbf{Surprisingly} , models trained on truncated datasets, retaining only the first half or fewer tokens, achieve comparable or even superior performance to those trained on full datasets. For example, a reward model trained on the Skywork-Reward-Preference-80K-v0.2 dataset outperforms the full dataset when trained on a 40% truncated dataset. This pattern is consistent across multiple datasets, suggesting the widespread presence of \textit{shallow preference signals} .We further investigate the distribution of the reward signal through decoding strategies. We consider two simple decoding strategies motivated by the shallow reward signal observation, namely Length Control Decoding and KL Threshold Control Decoding, which leverage shallow preference signals to optimize the trade-off between alignment and computational efficiency. The performance is even better, which again validates our hypothesis.The phenomenon of \textit{shallow preference signals} highlights potential issues in LLM alignment: existing alignment methods often focus on aligning only the initial tokens of responses, rather than considering the full response. This could lead to discrepancies with real-world human preferences, resulting in suboptimal alignment performance.

## introduction aligning large language models (llms) with human preferences is a core challenge in artificial intelligence (ai) research @xcite . preference datasets @xcite @xcite have played a critical role in addressing this challenge by capturing human judgments of model outputs. these datasets enable the identification and prioritization of responses that are more aligned with human expectations. preference-based optimization techniques, such as reinforcement learning with human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite , rely on these datasets to refine the decision-making process of models. despite the promise of these methods, there are several challenges associated with them. recent work @xcite @xcite has highlighted that reward models trained using rlhf may suffer from reward hacking. factors such as response format, length, and even the inclusion of emojis can influence quality judgments, resulting in potential inaccuracies. in this paper, we introduce a previously underexplored aspect of preference data. specifically, we observe that the signal indicating the superiority of the chosen response over the rejected one is not uniformly distributed across the entire response. in many cases, the relative quality of responses can be determined from only the early portion of the response-or even just a few tokens-rather than requiring an evaluation of the entire response. we refer to this phenomenon as shallow preference signals. this observation suggests that preferencebased optimization methods may not need to rely on the full response to effectively capture the distinguishing features of higher-quality responses. we hypothesize that focusing on the early portion of the response allows models to capture the most salient preference signals, resulting in more efficient training and potentially improved alignment performance. to test this hypothesis, we introduce a methodology where preference data is truncated at various positions, and models are trained on these truncated datasets. we analyze the distribution of preference signals in response pairs and conduct systematic experiments to validate the hypothesis that models trained on truncated preference data perform comparably to models trained on the full dataset. this is confirmed for both reward models and models fine-tuned with dpo. our findings demonstrate that the distinguishing features between the chosen and rejected responses are concentrated in the early part of the response. in fact, models trained on truncated datasets-using only the first half or fewer tokens of each response-achieve similar, or even superior, performance compared to those trained on the full dataset. for instance, a reward model trained on the skywork-reward-preference-80k-v0.2 @xcite dataset achieves an accuracy of only 75.85% on rewardbench @xcite . however, when the dataset is truncated to 50% and 40%, the accuracy increases to 75.88% and 76.35%, respectively. even with a truncation to 25%, the accuracy remains at 69.92%. similarly, a reward model trained on the rlhflow-pair-data-v2-80k-wsafetyrlhflowpair-data-v2-80k-wsafety 1 dataset achieves an accuracy of 65.96% on rewardbench. after truncating the dataset to 50% and 40%, the accuracy improves to 72.16% and 69.71%, respectively, with accuracy remaining at 62.44% for a 33% truncation. furthermore, our experiments suggest that the shallow preference signal phenomenon significantly impacts llm content generation. based on this observation, we find that simple strategies can perform well without needing complex decoding approaches. recent work @xcite @xcite has proposed various decoding strategies, but our findings indicate that by focusing on the early portion of the response, we can achieve 1 https://huggingface.co/datasets/rlhflow/pair_ data_v2_80k_wsafety an optimal trade-off between reward and kl divergence. to test this, we explore two decoding strategies-length control decoding and kl threshold control decoding-to see if the early-token bias observed during training affects generation at inference time. our results show that the differences between the dpo model trained on full preference data and the reference model are most noticeable in the early tokens of the generated response. as more of the response is generated, the difference decreases. this suggests that the reward signal in dpo training is concentrated in the early tokens, rather than being evenly distributed. @xcite also explores token distribution differences between base llms and aligned models, though their method primarily focuses on in-context learning, avoiding parameter fine-tuning. meanwhile, the findings of this paper may shed light on existing problems in llm alignment. our experiments validates that current alignment methods often focus on aligning earlier tokens, rather than considering full sentences. the latter portions of answers generated by llm tend to be generated through an auto-regressive mechanism, which does not exhibit significant quality variation through our decoding experiments. through extensive experiments, we validate our hypothesis that focusing on the early portion of the response allows models to capture the most salient preference signals, resulting in more efficient training and potentially improved alignment performance. however, alignment with truncated data is shallow alignment which only improves the performance on metrics but may keep further away from the realworld alignment with human values. @xcite proposes a related issue, but their work is confined to safety alignment and does not extend to the broader alignment challenges present in llms. instead, our work validates the phenomenon more systemically and extensively. in summary, the main contributions of our paper are as follows: ## methodology in this section, we introduce the methodology used to investigate the structure and front-loaded nature of reward signals in large language models (llms) trained with preference data. ## experiment: kl divergence and reward-kl tradeoff for evaluating response quality this section presents a set of experiments that examine the relationship between the kullback-leibler (kl) divergence between the dpo model and the reference model, and the reward-kl tradeoff during response generation. these experiments aim to validate the hypothesis that the reward signal in preference datasets is primarily concentrated in the early part of the response, highlighting the phenomenon of shallow preference signals. ## conclusion we introduce shallow preference signals, where key distinguishing features between preferred and non-preferred responses are concentrated in early response tokens. our experiments show that models trained on truncated data-retaining 40% to 50% of tokens-perform similarly or better in reward modeling and direct preference optimization (dpo) than those trained on full-length data. additionally, we highlight the limitation of current methods that focus mainly on initial tokens, suggesting the need for strategies that consider entire responses for more accurate alignment with human preferences.