When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning (2025)

Abstract: Existing work has shown that o1-level performance can be achieved with limited data distillation, but most existing methods focus on unidirectional supervised fine-tuning (SFT), overlooking the intricate interplay between diverse reasoning patterns. In this paper, we construct r1k, a high-quality reverse reasoning dataset derived by inverting 1,000 forward examples from s1k, and examine how SFT and Direct Preference Optimization (DPO) affect alignment under bidirectional reasoning objectives. SFT on r1k yields a 1.6%–6.8% accuracy improvement over s1k across evaluated benchmarks. However, naively mixing forward and reverse data during SFT weakens the directional distinction. Although DPO can partially recover this distinction, it also suppresses less preferred reasoning paths by shifting the probability mass toward irrelevant outputs. These findings suggest that mixed reasoning data introduce conflicting supervision signals, underscoring the need for robust and direction-aware alignment strategies. Our code and data are available at: https://github.com/16demi/ReasonAlign-analysis.

## introduction recent studies show that large language models (llms) can achieve strong reasoning performance by distilling knowledge from a small set of high-quality examples @xcite . methods like s1 @xcite and limo @xcite demonstrate that with just 817 to 1,000 curated samples, a 32b model can match or surpass larger systems. however, these approaches focus mainly on single-direction reasoning-solving problems step by step from question to answer. as shown in figure 1 , a model may learn to compute the kinetic energy of a gas molecule from its temperature, but not the reverse: inferring temperature from energy. this narrow focus overlooks a core aspect of human cognition: the inherently bidirectional nature of reasoning. humans commonly engage in backward reasoning, particularly in goal-directed problem solving @xcite . rather than reasoning solely from premises to conclusions, people often begin with a desired outcome and work backward through intermediate steps to reach known facts @xcite . motivated by this cognitive insight, recent studies have begun to explore reverse or backward reasoning in llm. mathgenie @xcite utilizes reverse derivation paths to improve robustness on math word problems. iterative question composing @xcite constructs intermediate subquestions that align with goal-driven, backward-style planning. in optimization modeling, optibench @xcite promotes reflective and socratic-style reformulations that partially embody reverse reasoning principles. while promising, these approaches remain constrained to short-context reasoning or domain-specific tasks. this leaves open the broader question of whether backward supervision can enhance long chain-ofthought @xcite reasoning and generalize across diverse scenarios. to investigate this, we construct a high-quality dataset, r1k, by systematically inverting 1,000 forward reasoning examples from s1k @xcite . reverse questions and reasoning paths are generated with cost-efficient deepseek-r1 model @xcite , without the need for expensive data collection, cleaning, or selection procedures. fine-tuning on r1k yields an approximate 1.6%-6.8% improvement over s1k. to further study the interaction between mixed data, we conducted extensive experiments on their combined effects. we observe that sft on re-figure 1 : we begin with the s1k dataset (x f , y f ) and generate reverse questions x r , along with their corresponding reverse cots and answers y r . we then fine-tune student models using cross-entropy loss under three settings as comparison: forward-only data, reverse-only data, and a mixture of both. to enhance directional consistency, we apply dpo to encourage directionally aligned responses while suppressing misaligned ones. concurrently, we track the log probability of y f and y r across multiple fine-tuning stages to investigate the models' learning dynamics. ## experiments we conducted experiments on the deepseek-r1's s1k dataset, which consistently outperformed the gemini-based variant. as the s1k has already been curated with quantity, diversity, and difficulty, we did not apply an additional filtering process. we fine-tune qwen2.5-instruct models (7b and 14b) in two stages on 8 a800-80gb gpus. first, we applied sft with lora (@xmath0, α = 512). then, we performed dpo using the trl library (von werra et al., 2020), with deepspeed zero-3 (jacobs et al., 2023) and flash attention @xcite to reduce memory usage. we employ opensource lm-eval-harness @xcite , with gpt-4o-mini to evaluate the accuracy. ## conclusion we constructed a high-quality reverse reasoning dataset r1k and demonstrated its effectiveness in improving reasoning ability. we further investigate the effects of mixed data during multi-stage fine-tuning, underscoring the need for improved alignment strategies to support robust reasoning.