Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models (2024)

Abstract: In reasoning tasks, even a minor error can cascade into inaccurate results, leading to suboptimal performance of large language models insuch domains. Earlier fine-tuning approaches sought to mitigate this by leveraging more precise supervisory signals from human labeling, larger models, or self-sampling, although at a high cost. Conversely, we develop a method that avoids external resources, relying instead on introducing perturbations to the input. Our training approach randomly masks certain tokens within the chain of thought, a techniquewe found to be particularly effective for reasoning tasks. When applied to fine-tuning with GSM8K on Llama-2-7B, this method achieveda 5% improvement in GSM8K accuracy and a 10% improvement in GSM-IC accuracy over standard supervised fine-tuning with a few codes modified. Furthermore, it is complementary to existing methods. When integrated with related explicit data augmentation methods, it leads to improvements across five datasets of various augmentation methods, as well as two different base models. We further investigate the mechanisms behind this improvement through case studies and quantitative analysis, suggesting that our approach may provide superior support for the model in capturing long-distance dependencies, especially those related to questions. This enhancement could deepen understanding of the premises in questions and prior steps.

## introduction despite the significant advancements large language models (llms) have achieved across various tasks, they continue to encounter difficulties with multi-step reasoning problems (lightman * corresponding authors: xiting wang, rui yan, and yongbin li. to improve the reasoning ability, most training methods focus on acquiring supervised signals obtained through costly means, such as human annotation @xcite , generation of larger models @xcite @xcite , or self-sampling @xcite @xcite . these methods enable the models to learn specific trajectories by precisely outlining the reasoning processes. in contrast, our method illustrates that an entirely different approach is equally effective. instead of requiring more precise guidance to direct the generative process, our technique achieves comparable results by incorporating random noise into the reasoning steps. for reasoning tasks, which are generally considered to require precise understanding, we find that this noise-based method is surprisingly effective, indicating that the model might have enhanced its performance through a denoising process. we summarize our contributions as follows: first, we propose a simple and effective masked thought fine-tuning (mft) method for improving language model reasoning. the implementation of our method is simple: it requires only the substitution of specific tokens with a [mask] in the chain of thought. this is done while maintaining the same procedures as the standard supervised fine-tuning (sft). with this minor modification, mft produces results comparable to those of a more complex data generation pipeline (see second, we analyze our method from a regularization perspective and introduce two guiding principles for effective regularization. specifically, we propose a noise injection framework in which our method and many commonly employed regularization methods are special cases, such as dropout @xcite and synonym substitution @xcite . we conduct a comparative analysis and find that these regularization methods are effective only when two guiding principles are met: (1) a portion of the tokens in the reasoning path should be retained without noise addition; (2) for positions where noise is added, it is crucial to ensure these positions maintain as little semantics as possible (see sec. 3.3). third, we explore the underlying reasons for the effectiveness of our method by revealing its impact on learning dependencies. by conducting both quantitative analyses and case studies, we observed that this method demonstrates an enhanced dependency on the initial mathematical questions and earlier steps. this suggests a possible reason: instead of depending solely on the local information generated by the model itself from recent steps, which can lead to a higher likelihood of hallucination or misdirection as the sequence extends, our method utilizes more information from the problem statement and earlier steps. these sources are more reliable and less prone to errors. consequently, this strategy might reduce the risk of misunderstanding the problem and inconsistencies in reasoning (refer to secs. 2.2 and 3.4). ## related work mathematical reasoning of llm. to improve the mathematical reasoning ability through supervised fine-tuning, the approach generally involves leveraging larger models or manual efforts to collect high-quality data @xcite @xcite @xcite . metamath @xcite augments its dataset by rewriting mathematical queries and providing answers with the help of an llm. mammoth @xcite compiles diverse mathematical datasets for training, enriching them with chains of thought (cot) and programs of thought (pot) rationales for reasoning, resulting in the mathinstruct dataset. rft @xcite is a simplified offline method of standard online rl. it enhances its training dataset with accurate reasoning paths by generating and collecting them via an sft model. ## conclusion we propose masked thought fine-tuning as a regularization technique during fine-tuning. our findings indicate that mft not only boosts the reasoning performance but also improves the model's ability to capture long-distance dependencies. this study could pave the way for future research on regularization strategies for reasoning tasks and further exploration of the interplay between language dependency and reasoning capabilities.