Structure Trumps Size: Rethinking Data Quality for LLM Reasoning (2025)

Abstract: As domain-specific datasets continue to expand, Large Language Models (LLMs) have achieved significant improvements across various fields through supervised fine-tuning (SFT). However, is more data always better for model fine-tuning? Through a series of controlled experiments, we discover that dataset structure—rather than mere size—plays a decisive role in enhancing LLM reasoning capabilities. While existing methods acknowledge that good data quality can make training more efficient, they primarily rely on simple heuristic strategies and lack systematic, quantitative frameworks for evaluating data quality. To address this gap, we introduce MCSQ—the first multi-dimensional quantitative framework for reasoning data management. MCSQ rigorously evaluates and optimizes datasets along six orthogonal dimensions. Through comprehensive controlled experiments, we find that selectively incorporating “distorted” (model-disagreed) or “mismatched” (low-relevance) samples—which are typically discarded in traditional approaches—can outperform conventional “clean” data on certain advanced reasoning benchmarks. Our findings challenge traditional assumptions about data “quality” in LLM fine-tuning and provide actionable, quantitative guidance for efficient, structure-aware dataset management. The datasets and codes are both available at https://github.com/xuhu0115/MCSQ.

## introduction large language models (llms) have demonstrated impressive reasoning capabilities across diverse domains, yet complex multi-step reasoning remains a significant challenge @xcite . supervised fine-tuning (sft) on reasoning traces has emerged as a promising approach to enhance these abilities by directly * corresponding authors. reasoning-focused sft has gained significant traction in recent years, with numerous studies demonstrating its effectiveness across mathematical @xcite , scientific @xcite , and logical reasoning tasks @xcite . this approach has evolved from early work on step-by-step reasoning distillation @xcite to more sophisticated methods incorporating diverse reasoning patterns @xcite , planning algorithms @xcite , and multi-stage reasoning frameworks @xcite . recent research has shown that models finetuned on reasoning traces can achieve significant performance improvements on complex problemsolving benchmarks @xcite , often matching or exceeding the capabilities of much larger models. despite these advances, the field still lacks a systematic understanding of what makes reasoning datasets effective, with most work relying on intuition-based curation strategies rather than principled analysis of dataset characteristics. this knowledge gap manifests in several critical, unresolved questions that directly impact reasoning sft effectiveness. first, the relationship between data volume and reasoning performance remains unclear-while scaling laws suggest "more is better," recent work hints at diminishing returns or even performance degradation with excessive data @xcite . second, the optimal composition of reasoning domains and types is poorly understood; practitioners lack guidance on whether to prioritize breadth (covering many domains) or depth (focusing on specific areas). third, the impact of data quality variations-including imperfect reasoning traces or examples of marginal relevance-has not been systematically investigated, leaving dataset filtering decisions to rely on unvalidated heuristics rather than empirical evidence. these uncertainties point to a fundamental limitation in current approaches: the lack of a principled framework for analyzing and optimizing reasoning datasets beyond simple volume or binary quality metrics. while extensive research has explored data selection for general llm pretraining @xcite and instruction tuning @xcite , the unique requirements of reasoningcentric sft data demand specialized analysis tools that can capture the multi-faceted nature of reasoning quality. to address these challenges, we introduce the multi-dimensional quantitative framework for evaluating distilled chain-of-thought sft data quality (mcsq). grounded in information theory principles @xcite , mcsq decomposes reasoning dataset quality into six concrete, measurable dimensions: volume, scope, granularity, variety, distortion, and mismatch. through controlled experiments where we systematically vary one dimension while holding others constant, we uncover several counter-intuitive findings that challenge conventional wisdom in reasoning dataset construction. our key discoveries include: (1) data structure consistently outweighs volume in determining reasoning performance, with scaling beyond moderate thresholds yielding diminishing returns; (2) "imperfect" data can significantly boost performance on certain advanced reasoning tasks; and (3) the optimal balance between domain specialization and diversity depends critically on the target task. based on these findings, we make three major contributions to the field: ## related work research on llm reasoning has progressed from inference-time prompting-such as chain-of-thought @xcite , self-consistency @xcite , and searchbased methods @xcite -to approaches that embed reasoning capabilities directly into model parameters via targeted data and fine-tuning. we briefly review three relevant strands: (1) general data selection methods for llm training, (2) reasoning trace generation and fine-tuning, and (3) reasoning-centric data selection strategies. these areas collectively motivate the need for a principled, multi-dimensional approach to reasoning dataset construction. ## methodology in this section, we present our approach for quantitatively characterizing and manipulating reasoningcentric fine-tuning datasets. ## experiments the core objective of this work is to systematically investigate how distinct structural properties of reasoning datasets affect llm fine-tuning and performance. to this end, we conduct extensive experiments aimed at answering the following research questions: by systematically investigating these questions, our results(figure 3 ,appendix d) provide actionable insights into how each dataset property shapes llm reasoning, enabling principled and effective data curation strategies for advanced model development. ## discussion one concern is the reliability of distortion and mismatch metrics, as they rely on model-based judgments. to mitigate bias, we employed ensemble scoring (gemini, qwen-7b, qwen-32b) and discretized outputs into bins. furthermore, a human annotation study on 500 samples showed over 90another concern is the independence of mcsq axes. while objective information theory (oit) provides theoretical justification for their orthogonality, we acknowledge that residual correlations may exist in practice. our controlled sampling design minimizes confounding, and future work will quantify these correlations explicitly with dependency analysis. finally, we recognize the trade-off between reasoning specialization and general capability observed in our experiments. we propose a preliminary "three-stage filtering funnel"-macro (domain scope), core (challenge and relevance), and micro (granularity/diversity)-to balance reasoning gains with general knowledge retention. ## conclusion in this work, we present mcsq-a systematic, quantitative framework for dissecting and optimizing the data characteristics that underpin llm reasoning. by decomposing dataset composition into six interpretable, measurable dimensions-volume, scope, granularity, variety, distortion, and mismatch-we move beyond conventional, volume-centric heuristics and provide the first large-scale evidence that dataset structure decisively shapes reasoning performance. through controlled fine-tuning experiments, we reveal that targeted adjustments to data structure consistently yield greater improvements in llm reasoning than naive data scaling.moreover, even certain data containing 'noise' or exhibiting 'low relevance' can contribute to the enhancement of reasoning capabilities. importantly, our findings expose a robust specialization-generalization trade-off, highlighting the need for data-efficient and structure-aware curation strategies. our results challenge prevailing assumptions about data "quality" in llm fine-tuning and provide actionable guidance for dataset curation, with broad implications for model distillation, instruction tuning, and automated data selection. we believe that the mcsq framework paves the way for a new generation of reasoning-centric llms, inspiring further research into automated, principled, and task-aware dataset optimization.