Self-Training Elicits Concise Reasoning in Large Language Models (2025)

Abstract: Chain-of-thought (CoT) reasoning has enabled large language models (LLMs) to utilize additional computation through intermediate tokens to solve complex tasks. However, we posit that typical reasoning traces contain many redundant tokens, incurring extraneous inference costs. Upon examination of the output distribution of current LLMs, we find evidence on their latent ability to reason more concisely, relative to their default behavior. To elicit this capability, we propose simple fine-tuning methods which leverage self-generated concise reasoning paths obtained by best-of-N sampling and few-shot conditioning, in task-specific settings. Our combined method achieves a 30% reduction in output tokens on average, across five model families on GSM8K and MATH, while maintaining average accuracy. By exploiting the fundamental stochasticity and in-context learning capabilities of LLMs, our self-training approach robustly elicits concise reasoning on a wide range of models, including those with extensive post-training.

## introduction chain-of-thought (cot) reasoning has significantly improved the ability of large language models (llms) to perform complex tasks @xcite . the effectiveness of cot reasoning has been attributed to the additional computation allocated during inference, as each intermediate reasoning token enables the model to perform an additional forward pass through its parameters @xcite . on the other hand, this inherently incurs additional inference cost and latency, roughly proportional to the number of output tokens @xcite . we posit that current models often generate more tokens than necessary to accomplish the task, incurring extraneous inference costs. typical reasoning chains sampled from the original model typical samples from the original model distribution marked in blue, as shown in the blue box, contain redundant context as marked in gray. we fine-tune the model with relatively concise reasoning paths within the subregion of the model output distribution marked in orange. this elicits the model to reason more efficiently, as shown in the orange box, retaining only the essential parts while still leading to the correct answer. distribution, exemplified in the blue box of figure 1 , includes many tokens that do not contribute to the solution, specifically verbose explanations and repetitive phrasing. similar observations have also been made in prior work @xcite @xcite . we argue that the redundancy in reasoning chains of current models is not surprising. llms have not been explicitly trained to utilize intermediate tokens for reasoning in the most efficient man-ner. rather, cot reasoning is an emergent ability of llms @xcite first uncovered through novel few-shot prompting @xcite . posthoc analysis suggests that reasoning ability is derived from procedural knowledge in pretraining data @xcite , which is not optimized for conciseness. therefore, it is natural that the default behavior of llms does not optimize for efficient use of tokens for reasoning. however, this does not preclude the possibility that llms possess the ability to reason more efficiently. indeed, the distribution of reasoning path lengths shown in figure 1 reveals a compelling insight: the presence of shorter, more efficient reasoning paths within the model's output distribution, marked in orange, suggests a latent capacity for concise reasoning. this observation motivates our central hypothesis: that by selectively leveraging these existing concise examples, we can fine-tune the model to elicit this latent ability and shift its default output distribution towards more efficient reasoning. in this paper, we propose a simple yet effective fine-tuning method to elicit efficient, concise reasoning in models for a given target task. we note that existing zero-shot prompting methods fail to reliably elicit concise reasoning, exhibiting near negligible impact on task-specialized models (section 2.2). instead, we can leverage best-of-n (bon) sampling and few-shot conditioning (fs) to reliably generate concise training data (section 3). we can then apply standard fine-tuning to distill the length-reducing benefits of bon sampling and fewshot prompting back into the model itself while avoiding inference-time overheads. across gsm8k and math datasets and a wide variety of model families, our unified few-shot conditioned best-of-n sampling (fs-bon) method achieves a substantial reduction in output length of 30% on average, a 2.4x improvement over previous fine-tuning baselines @xcite , while preserving overall accuracy. our analysis shows that trained models adaptively adjust output length based on question complexity, preserving detail for difficult problems while simplifying responses to easier ones. we confirm that these results remain consistent across various model scales and diverse reasoning domains such as business, chemistry, and physics. this shows that fine-tuning with carefully curated, self-generated data can effectively unlock latent concise reasoning abilities within llms, leading to significantly more costeffective inference for complex tasks. 20 40 60 80 100 120 normalized reasoning length (%) probability density llama-3.2-3b deepseekmath-7b gemma-2-2b qwen2.5-3b qwen2.5-math-1.5b figure 2 : models already exhibit relatively concise reasoning in their output distributions to differing degrees. distribution of output token lengths in reasoning paths of gsm8k, normalized by the average length of correct reasoning paths for each question, across five main models. reasoning paths to the left of the gray line are relatively shorter than the model's default output (section 2) ## preliminary investigation in this section, we probe the potential of current models to perform concise reasoning (section 2.1) and investigate whether zero-shot prompting can reliably elicit such behavior (section 2.2). we start with a definition of concise reasoning and key experimental setup. ## method to address the limitations of zero-shot prompting, we propose a simple yet robust fine-tuning (ft) approach to reliably elicit concise reasoning in llms while preserving accuracy. we focus on self-training approaches, to unlock and refine the latent concise reasoning abilities of current llms, observed in section 2.1. self-training not only removes external dependencies, but we posit that it helps preserve reasoning capability, as the training data originates from the model's own distribution. ## experimental setup models to account for realistic task-specific deployment settings, we select recent moderately sized post-trained models. we also consider mathspecialized models to evaluate on models that have been optimized for specific task domains. we consider five main models for our key experiments: llama-3.2-3b @xcite , gemma-2-2b @xcite , qwen2.5-3b @xcite , qwen2.5-math-1.5b @xcite , and deepseekmath-7b @xcite . we investigate scaling on llama-3.2-{1b, 3b} and llama-3.1-8b. tasks and datasets we focus on challenging reasoning tasks where (1) cot reasoning significantly improves model performance, (2) only the final answer is relevant, and (3) models achieve moderate performance. reasoning length reduction is desirable under the first and second conditions as it can reduce inference latency without affecting utility. the third condition is necessary to assess accuracy preservation. we consider two mathematical reasoning datasets: gsm8k @xcite and math @xcite , where the models achieve accuracies of 40-90% and 20-70%, respectively. for evaluation, we utilize the test set of the gsm8k and math500 dataset @xcite . we explain details in section c.1. evaluation metrics we evaluate methods using two primary metrics: accuracy and length. accuracy is evaluated using python-based parsing code, described in section c.2. length is defined as the average number of output tokens in all reasoning paths, including incorrect ones, as output tokens incur inference costs in deployment scenarios regardless of their correctness. we focus on number of output tokens for clear comparison, as number of input tokens are similar across methods, and output tokens affect wall-clock latency to a higher degree @xcite . we further justify this choice in section c.3. we also employ relative accuracy and relative length metrics to better evaluate how well each method elicits concise reasoning while maintaining accuracy. specifically, relative accuracy is the ratio of the given method's accuracy to the baseline accuracy, and relative length is the ratio of the given method's average length to the baseline average length, using a strong baseline zero-shot prompt @xcite . we use greedy decoding throughout evaluation to ensure reproducibility foot_1 . baseline methods for baselines, we consider zero-shot prompting (section 2.2) and fine-tuning directly on ground-truth answers, and with external supervision from human-and gpt-4o-generated concise reasoning paths. for existing fine-tuning methods on concise reasoning, we reproduce rational metareasoning (rm) @xcite , which follows a similar approach to our naive bon method but with two key differences: (1) a reward function that balances efficiency and utility (i.e., output length and accuracy), and (2) iterative training via expert iteration @xcite . ## discussion default reasoning behavior of llms previous research has shown that the cot reasoning ability of llms originates from procedural knowledge in pretraining data @xcite . modern llm training pipelines utilize high-quality math, code, and synthetic reasoning data to enhance reasoning, but these do not promote conciseness @xcite . furthermore, recent 'thinking' models are reinforced to use additional tokens to improve reasoning performance, rather than save on token budget (openai, 2024). thus, current llms naturally exhibit redundant reasoning. we believe that incorporating concise reasoning supervision or rewards in training pipelines can be beneficial for model efficiency, especially for 'thinking' models with lengthy internal reasoning. ## conclusion we tackle redundancy in cot reasoning by hypothesizing that llms possess a latent capacity for concise reasoning, as evidenced by shorter correct reasoning paths within their output distributions. we introduce fine-tuning methods that leverage self-generated data from best-of-n sampling and few-shot conditioning to elicit this capacity. our fs-bon method significantly reduces reasoning length by 30% while maintaining accuracy. this implies fine-tuning with curated self-generated data can reliably unlock latent concise reasoning, enabling more efficient inference.