Language Models Can Easily Learn to Reason from Demonstrations (2025)

Abstract: Large reasoning models (LRMs) tackle complex problems by following long chain-of-thoughts (Long CoT) that incorporate reflection, backtracking, and self-validation. However, the training techniques and data requirements to elicit Long CoT remain poorly understood. In this work, we find that language models can effectively learn Long CoT reasoning through data-efficient supervised fine-tuning (SFT) and further parameter-efficient low-rank adaptation (LoRA). Crucially, we find that the structure of Long CoT is critical to the learning process in this data-efficient fine-tuning process. Training on content-incorrect examples, e.g. those lead to incorrect answers or corrupted digits, still leads to significant performance gains. In contrast, training on structurally incorrect examples, e.g., with shuffled or deleted reasoning steps, yield smaller improvements or even degrade performance.

## introduction large reasoning models (lrms) leverage long chain-of-thoughts (long cots) with reflection, backtracking, and self-validation to tackle challenging reasoning tasks @xcite @xcite . however, training llms to elicit long cots remains an open problem, as existing methods are either closed-sourced @xcite or expensive to replicate @xcite . in this paper, we first show that an llm can be easily taught to produce long cot responses, significantly improving its reasoning capabilities. in particular, we find that this learning process can be both data-efficient and parameter-efficient. by performing fully supervised fine-tuning (sft) deepseek-r1 with either lora or full-parameter update, the model learns to perform reflection and backtracking by using keywords such as "however" and "alternatively". consequently, the fine-tuned models improve significantly on five math and coding benchmarks (olympiad. and lcb short for "olympiadbench" and "livecodebench"), matching openai o1-preview performance. this shows that learning to reason can be data-and parameter-efficient. figure 2 : sft results on structure or content perturbed training data of qwen2.5-32b-instruct. performance is averaged over math 500, aime 24, amc 23 and olympiadbench. the construction procedure of these traces can be found at §4. we find that training on structurally perturbed traces (e.g., shuffled, inserted, or deleted steps) leads to significantly worse performance than training on content-perturbed traces (e.g., incorrect final answers, corrupted digits, or missing keywords). this highlights the critical role of long cot structure in learning to reason from demonstrations. with only 17k samples generated by deepseek r1, the qwen2.5-32b-instruct model achieves performance competitive with openai o1-preview across a wide range of math and coding tasks @xcite @xcite . in particular, it achieves 90.8% in math-500 (+6.0%), 56.7% in aime 2024 (+40.0%), 85.0% in amc 2023 (+17.5%), 60.3% in olympiadbench (+12.7%) and 57.0% in livecodebench (+8.1%) @xcite . even further, the model can achieve o1preview performance by updating fewer than 5% parameters with lora fine-tuning @xcite . we show that the model successfully learns to reflect and revise its intermediate thoughts (e.g., frequently using reasoning keywords such as "alternatively" and "wait, but") and adopts long, coherent cots to tackle challenging problems (fig. 1 ). crucially, we identify the long cot structure as the key characteristic of distilled data for eliciting strong reasoning performance. we conduct two sets of controlled studies by altering either the content of individual reasoning steps or the overall logical structure. to alter content, we perturb samples by replacing numbers with random digits or deleting reasoning keywords. surprisingly, we find that these perturbations have little impact on the model performance: even when 50% of numbers in training samples are randomly changed, the model only observes 3.3% lower accuracy on the most challenging math benchmark, aime 2024, as compared to training with correct samples. to alter the long cot structure, we separate responses into reasoning steps and randomly shuffle, insert, or delete these steps. we observe that the trained model is sensitive to such structural perturbations that break logical coherency in the long cot. for example, when 67% of the training samples' reasoning steps are shuffled, accuracy drops by 13.3% on aime 2024 problems relative to training with correct samples. our key contributions are: training to improve reasoning capabilities of llms llm reasoning capabilities can be improved by approaches such as iterative selfimprovement and reinforcement learning (rl) @xcite @xcite @xcite . in particular, @xcite shows that incorrect data can enable efficient reasoning capability training. more recently, tulu-3 introduces reinforcement learning with verifiable rewards (rlvr) to improve performance in tasks such as math and coding @xcite @xcite . prime proposes a rl-based method without process labels @xcite . the recent release of deepseek r1 @xcite demonstrates that llms can learn to produce long cot and improve reasoning using a pure rl-based approach. instead of bootstrapping reasoning ability, this paper focuses on the surprising data-and parameter-efficiency of distilling reasoning abilities from an existing reasoning model to an llm. we find that training on incorrect data can also enhance llm reasoning, but unlike @xcite , we explore this in sft instead of rl. distillation distilling the outputs or logits generated by a larger or more capable model has become a standard technique to enhance model performance @xcite . typically, responses generated by higher-quality models are used to perform supervised fine-tuning on smaller models @xcite . the vicuna model, for instance, demonstrates that chatgpt-generated responses can be used to effectively and cheaply distill highquality chatting capabilities @xcite . in this paper, we show that reasoning capabilities can also be distilled. we note that concurrent work has also observed similar trends in distilling reasoning capability @xcite @xcite . our paper differs from these recent works by demonstrating that reasoning distillation can be achieved efficiently with minimal parameter updates, and the key role of the structure of long cot in the process. ## experiments setup we describe the experiments settings for §4. distillation we use @xcite and qwq-32b-preview @xcite , two open-source models with reasoning capabilities, to generate our distillation data. we select difficult prompts from the amc/aime foot_0 , math, and olympiad subset from the numina-math dataset @xcite , as @xcite implies that hard problems can improve performance. we also incorporate coding problems from apps @xcite and taco @xcite datasets. specifically, we use gpt-4o-mini to classify the difficulty of the prompts according to the aops standard @xcite , and select math problems of difficulty higher than leval 3, olympiad higher than level 8, and all aime/amc problems. we verify the correctness of the traces by checking against ground truth solutions using exact matching for math problems and code execution for coding problems. in total, we curated 12k math and 5k coding problems with correct responses from qwq to serve as our training data. for r1 samples, we directly use the public r1-17k reasoning dataset 2 that is curated following a similar procedure. training details we perform training using llama-factory @xcite . we train the qwen2.5-32b-instruct using a batch size of 96, learning rate 1e-5 with a warm-up ratio of 0.1 and linear learning rate decay @xcite , following similar hyperparameters in @xcite . we use the next token prediction loss as the training objective @xcite . we use the same hyper-parameters except a 1e-4 learning rate for lora fine-tuning. the full fine-tuning experiment is done with 8xh100 gpus in 20 hours. ## language models can easily learn to reason from demonstrations in this section, we first show that a small amount of well-curated data, along with a simple parameterefficient fine-tuning method (e.g., lora), can effectively improve reasoning capabilities in a large language model ( §4.1). then we present our observation that the long cot structure is a key to the learning process ( §4.2). ## ablation studies in this section, we conduct a series of ablation studies to answer the following questions: ## conclusion large reasoning models unlock new capabilities by using a longer chain of thoughts that involves reflection and backtracking to answer challenging problems. in this paper, we show that such capability can be easily fine-tuned using a few thousand examples and with low-rank adapters. we further show that the long cot structure of training examples is a key to the learning process. finally, we discuss several ablations with various teacherstudent models and compare them to the best-of-n approach. to solve the problem, we start by analyzing the given expression and the conditions under which it represents a real number. the expression is: for the square root to represent a real number, the argument of the square root, which is the fraction inside, must be nonnegative. additionally, since we are dealing with logarithms, n must be a positive integer. let's break down the solution into detailed steps: 1. condition for the denominator the denominator of the fraction is log n -3. for the fraction to be nonnegative, we need log n -3 > 0. this implies that log @xmath0, which translates to @xmath1. therefore, n must be greater than 1000.