Socratic Style Chain-of-Thoughts Help LLM s to be a Better Reasoner (2025)

Abstract: Synthetic data generation has emerged as a promising approach to enhance the reasoning capabilities of large language models. However, existing methods remain hindered by high costs—either through expensive API access or additional intermediate training—and are limited in their ability to generalize across different domains. To address these challenges, we propose a multi-agent debate framework based on the Socratic questioning strategy, abbreviated as SoDa. Distinguished from previous methods that prioritize data quantity, we highlight the wisdom of Socratic questioning in augmenting reasoning quality by deepening the thinking process to encourage exploration and broadening it to motivate self-reflection on each question. Combined with our efficient production pipeline, SoDa enables scaling while maintaining affordable costs. We use SoDa to generate diverse datasets for mathematics and code generation tasks with the Qwen2.5-7B-Instruct model, successfully fine-tuning a range of foundation models, from general-purpose ones to OpenAI o1-like ones. For mathematics, the experimental results show that SoDa outperforms the performance of existing datasets at the same scale, achieving improvements ranging from 1.3% to 13.5%. Remarkably, SoDa with 30K examples even surpasses the ScaleQuest dataset with 1000K samples, demonstrating significant efficiency. Our findings highlight the potential of SoDa as a universal, scalable, and cost-effective method for enhancing reasoning capabilities in large models across domains.

## introduction the enhancement of reasoning capabilities in large language models (llms) has become a recent research focus @xcite . recent studies have achieved notable progress by utilizing large language models to synthesize high-quality training data @xcite , resulting in substantial performance improvements on mathematical reasoning benchmarks such as gsm8k @xcite and olympiad-bench @xcite . however, these methods depend on strong supervisory signals from two primary sources: i) powerful llms, which generally incur high api costs @xcite , and ii) finetuned smaller models, which demand extensive training and struggle with domain adaptation @xcite . these dependencies inherently constrain the scalability and flexibility of their data generation processes. furthermore, existing approaches are predominantly tailored to mathematics, with limited generalizability to other domains such as code generation. consequently, there is an urgent need for a cost-effective and adaptable method for generating high-quality data across diverse domains. we draw inspiration from the socratic questioning method @xcite , a systematic approach to critical thinking that uncovers under-lying assumptions, clarifies concepts, and motivates deeper understanding through iterative questioning. this iterative process mirrors the process of human learning, which involves experiencing and exploring through trial-and-error @xcite interactions with the environment, ultimately leading to improved solutions. if models can adopt this paradigm, it would enable a more natural extension of existing data-driven methods. this approach fundamentally differs from the direct answer-generation style of current chain-ofthoughts (cot) methods and offers several key advantages. first, it does not rely heavily on advanced models, making it highly efficient for scaling. second, it requires no additional training and instead relies solely on self-debating, allowing generalization across diverse domains and even real-world problem-solving scenarios. ultimately, the trained models would demonstrate enhanced adaptability and robustness, paving the way for broader and more versatile applications. we propose the soda (socratic debate), a model debate framework based on the socratic questioning strategy @xcite . for the socratic component, we incorporate its principles to iteratively explore and reflect, while additional constraints ensure comprehensive generation of thoughts and answers. for the debate component, we simplify the process by reducing it to a fixed number of turns between two roles (i.e., the socratic and the student), enabling efficient execution without compromising quality. to further accelerate scaling, we optimize the generation pipeline by leveraging the observation that in-context learning (icl) can effectively guide the debate process. this allows us to generate high-quality datasets using a 7b model, qwen2.5-7b-instruct @xcite , which significantly reduces computational costs. ultimately, our method achieves efficient dataset generation and maintains adaptability across diverse domains (see based on the proposed soda, we expanded existing datasets by converting them into socraticdialogue formats. this includes three math datasets: dart-math @xcite , numi-namath @xcite , and scalequest @xcite , as well as a code dataset mceval-instruct @xcite . for math tasks, training with the expanded dataset resulted in significant performance improvements on llama-3.1-8b @xcite , with gains ranging from 1.3% to 13.5%. these improvements were consistently observed across various models, such as qwen2-math-7b @xcite , a mathspecific model, and skywork-o1 (skywork-o1, 2024), an o1-like model. notably, utilizing 30k soda-enhanced data achieves comparable or superior performance to full-scale datasets (see figure 1 ). additionally, our method achieved an average 2.7% improvement on code tasks. these results suggest that our approach enhances cot reasoning, making it a versatile and efficient method for synthetic data generation across diverse domains.. ## methods we present a general framework (see figure 2 ) for efficiently generating synthetic conversational socratic-style chain-of-thought (cot) data, which includes two key components: the socratic module, designed to generate insightful questions that stimulate critical thinking, and the multi-agent debate module, which extends the dialogue into multi-turn interactions for deeper exploration. to accelerate the data generation process, we propose an efficient pipeline leveraging in-context learning with meticulously curated high-quality demonstrations. ## conclusion in this paper, we focused on enhancing data synthesis frameworks to achieve high-quality generation while maintaining low cost. to this end, we proposed soda, a multi-agent debate framework grounded in the socratic questioning methodology. we initialized the debate roles, the socratic and the student, using curated prompts based on fundamental socratic principles, facilitating a structured and iterative reasoning process. to reduce generation costs, we optimized the pipeline by leveraging in-context learning with smaller models instead of relying on advanced llms, ensuring scalability without compromising quality. after fine-tuning a wide range of foundation models, our approach outperformed existing methods at similar data scales. notably, it even surpassed numina-math and scale-quest datasets, which contains 20x more data, on average benchmarks. this work highlights the potential of leveraging small llms to enhance powerful models effectively and cost-efficiently.