Enhanced Data Synthesis for LLM through Reasoning Structures Generated by Hierarchical GF low N et (2025)

Abstract: Large language models (LLMs) excel in problem-solving but require training data with diverse reasoning processes. Existing methods mainly optimize instruction-response pairs but lack a systematic design for the underlying reasoning structure. This paper proposes RSS: a Reasoning Structure driven data Synthesis method. We first proactively develop a hierarchical GFlowNet to construct reasoning structures efficiently through a coarse-to-fine directed acyclic graph (DAG) growth process. Then reasoning DAGs are leveraged to actively guide the instruction generation via an iterative suggester-editor workflow and enhance response quality using a structure-aware strategy. Experiments show that LLMs trained on our synthetic datasets achieve 48.50%, 84.00%, 79.90% for AlpacaEval2, GSM8K and HumanEval, outperforming existing data synthesis methods.

## introduction large language models (llms) have demonstrated human-like problem-solving abilities across a wide variety of tasks @xcite @xcite . the reasoning process in problem-solving is often represented as graph-like reasoning structures @xcite . training llms to master diverse reasoning structures is essential to unlock the abilities of llms for more general and complex tasks @xcite . however, collecting high-quality data with rich reasoning process is both costly and laborintensive @xcite . to address this issue, llm-driven approaches are proposed to generate synthetic data @xcite @xcite . some approaches leverage llms to directly augment seed data using prompts, such as self-instruct and evol-instruct @xcite . in these methods, the inherent reasoning structures to response are indirectly altered by augmented questions, as illustrated in figure 1 (a). subsequently, methods in figure 1 (b) establish a pool of tags (knowledge points, skills, or personas) and use llms to generate data based on the combination of these tags @xcite @xcite . however, this combination influences reasoning structures in a passive manner, as logical connections among tags are not considered. additionally, studies in figure 1 (c) focus on enhancing response quality from existing data via multi-query prompting frameworks like tree-of-thoughts (tot) and monte carlo tree search (mcts) @xcite . these methods inefficiently use trialand-error to find accurate solutions, lacking guidance from the original questions' reasoning structures. therefore, we propose a reasoning structure driven data synthesis method, abbreviated to rss. it proactively generate reasoning structures as direct acyclic graphs and uses them to synthesize instructions and responses, as illustrated in figure 1 @xcite . with explicit guidance from reasoning structures, we can generate logical, real-world-like questions and high-quality responses efficiently. synthesizing diverse reasoning structures with complex logical connections is challenging due to the vast combination space created by the diversity of real-world questions @xcite . to address this, we propose a three-level hierarchical cognitive schema (h-schema), inspired by the chc theory @xcite , which includes abstract-level, competency-level, and examplelevel cognitive nodes for representing problemsolving capabilities (see figure 2(a) ). reasoning structures are modeled as directed acyclic graphs (dags) at each level, linking cognitive nodes through logical dependencies. nodes at higher levels define candidate child nodes for finer levels, leveraging dependencies to reduce the search space. these hierarchical relationships provide an organized framework that enhances efficiency and effectiveness of the search process. the gflownet excels at generating diverse dags through a stepwise growth process @xcite , adding nodes iteratively. we developed a hierarchical gflownet (h-gflownet) to sample reasoning dags by connecting nodes from h-schema (figure 2(b) ). at each generation step of h-gflownet, three policy networks corresponding to generating dag for each level of h-schema, sequentially predict the newly added node under the condition the sampled node at higher level. the policy networks undergo end-to-end a reinforcement learning training paradigm with feedback on dag faithfulness as evaluated by llms, with guidance provided by the reward prompt. starting with reasoning dags, we synthesize instructions and responses as shown in figure 2(c ). unlike existing methods, we use a prior-planned reasoning structure for direct control over the question's reasoning process. for instruction generation, we parse the dag into a topological noderelation sequence and use a suggester-editor workflow, similar to @xcite , that refines the question step by step to accurately reflect the reasoning structure and ensure high quality. reasoning structures have proven beneficial for producing high-quality responses. thus, we use a rea-soning structure-aware strategy for response. by integrating the reasoning dag into methods like plan-and-solve @xcite , tot @xcite , and mcts @xcite , we reduce the cost of constructing these structures. since the reasoning dag inherently represents the problemsolving structure, a straightforward plan-and-solve strategy @xcite can generate effective responses guided by our reasoning dag. we validate that our method outperforms baseline methods on mainstream tasks, such as instruction-following, math, and code. additionally, we find that starting with complex and rich reasoning dags is an effective way to generate training data with high-quality. ## method reasoning structure as foundation the reasoning structure, capturing human cognitive processes during problem-solving @xcite , serves as our framework's cornerstone. building upon thought chain formalizations in got @xcite and dot @xcite , we formalize reasoning processes as directed acyclic graphs (dags) where nodes represent atomic problem-solving subprocedures and edges encode stepwise logical dependencies. this dag abstraction illustrates the compositional nature of reasoning process. the following sections describe our methodology for (1) efficiently sampling dags from combinatorial spaces and (2) converting these graphs into instruction-response pairs while preserving logical integrity. ## related works we summarize existing synthetic data generation approaches as follows: seed transformation-based methods, such as self-instruct and alpaca, evol-instruct @xcite @xcite , use llms to transform seed questionanswer pairs through evolving/rephrasing/backtranslation and related techniques @xcite @xcite ), yet they face limitations in data diversity due to a constrained number of prompts. in parallel, research on enhanced response generation emphasizes explicit reasoning processes, with innovations like star @xcite and rft @xcite using iterative verification and reject sampling to refine answers. while more recent works actively explore multiple reasoning paths via searching such as tot, rest-mcts and rstar @xcite @xcite . however, these methods encounter issues with inconsistent reasoning flows and high computational costs. abstract representation approaches, such as personabased generation @xcite and hierarchical taxonomies @xcite @xcite , aim for scalable instruction synthesis but often produce shallow reasoning structures due to random combination of representations. recently, graph-based solutions such as mathscale and kpsmath @xcite primarily encode topic statistics or focus on topic co-occurrence. alphageometry use theorem-proof dags for geometric problem generation @xcite . different from previous works, our approach is a data synthesis method driven by general and complex reasoning structures generated by a hierarchical generative model called h-gflownet. ## conclusion this study addresses the challenge of generating synthetic data with complex reasoning capabilities for the fine-tuning of llm through reasoning structure design. by shifting from instruction-response optimization to active reasoning structure construction, we propose a framework that integrates hierarchical cognitive modeling and efficient combinatorial sampling. experiments across instructionfollowing, mathematical reasoning, and code generation benchmarks validate that complex reasoning dags significantly enhance the quality and controllability of synthetic data compared to existing data synthesis strategies.