Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning (2025)

Abstract: When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles—specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format)—differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).

## introduction recently, there has been a surge of enthusiasm in researching agents based on large language models (llms) @xcite , with the aim of achieving human-level artificial intelligence or beyond. despite llms showcasing remarkable capabilities in various areas, they have not inherently demonstrated strong agent capabilities, such as multi-step reasoning @xcite @xcite and tool use @xcite @xcite . this shortfall has directed significant attention toward incorporating datasets tailored for agent tasks to enhance the agent capabilities of llms (chen et al., 2023; zeng et al., 2023; chen * work was done during the internship in alibaba group. ## methodology in this section, we first introduce the shuffle-aware discriminator (shad), which is proposed to adaptively distinguish between reasoning and boilerplate tokens. we then discuss how to develop our reasoning-highlighted fine-tuning (rft) based on the discrimination results. ## experiments we now present experiments to evaluate the effectiveness of our method in enhancing llms' agent capabilities, particularly in multi-step planning and tool usage, for solving complex real-world problems. we begin by detailing the experimental setup, followed by the analyses of the results. ## analysis on shad and rft in this section, we first present a case study on the effectiveness of shad in distinguishing different tokens, followed by a comprehensive analysis of how rft functions. ## conclusion in this paper, we highlighted the importance of distinguishing between reasoning and boilerplate tokens and introduced a shuffle-aware discriminator (shad) to automatically achieve this. building on shad, we further developed a new reasoning-highlighted fine-tuning (rft) method to enhance reasoning learning during llm fine-tuning, thereby improving agent capabilities. extensive results demonstrated that our method significantly enhances llms' ability to solve complex real-world problems. in the future, we plan to extend our approach to the entire sft domain and develop more refined mechanisms, such as token-level reweighting, to better leverage our token differentiation results.