C oncise RL : Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models (2025)

Abstract: Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-free conciseness score used as a reward signal within a reinforcement learning framework to guide models toward generating correct and concise reasoning traces. This score is evaluated by a large language model acting as a judge, enabling dynamic, context-aware feedback beyond simple token length. Our method achieves state-of-the-art efficiency–accuracy trade-offs on the MATH dataset, reducing token usage by up to 31x on simple problems while improving accuracy by 7%, and on the hardest problems, it outperforms full reasoning by +7.5% accuracy with up to 3.6x fewer tokens. On TheoremQA, our method improves accuracy by +2.2% using 12.5x fewer tokens. We also conduct ablation studies on the judge model, reward composition, and problem difficulty, showing that our method dynamically adapts reasoning length based on problem difficulty and benefits significantly from stronger judges. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/ConciseRL.

## introduction large language models (llms) have recently made significant progress in solving complex multi-step tasks, such as mathematics, code generation, and symbolic reasoning. reasoning models such as o1 @xcite , @xcite , and s1 @xcite are trained to reason explicitly through intermediate steps, often using reinforcement learning (rl) to optimize for correctness and structural fidelity. while this explicit reasoning improves accuracy and interpretability, it also introduces a major challenge: reasoning traces tend to be excessively long @xcite . these models frequently continue reasoning well past the point where the correct answer is reached, resulting in wasted computation, degraded readability, and sometimes even contradictions or hallucinated steps. addressing this problem is essential for reducing inference costs and improving the usability of llm reasoning in real-world applications. in this work, we propose a novel approach to reduce excessive reasoning in llms by teaching them to generate answers that are both correct and concise. our method introduces a semantically informed conciseness score as a reward function for training reasoning models and an llm as a judge to evaluate the conciseness of the reasoning trace. one key insight is that while accuracy can often be judged deterministically (e.g., matching answers or symbolic execution), conciseness is inherently subjective and better suited for llm-based evaluation. previous work has explored llm-as-a-judge setups for evaluating factual accuracy, style, and helpfulness @xcite , but existing methods for controlling reasoning length rely on token count or static heuristics, which fail to capture semantic efficiency. our method goes beyond simple token count: concise traces are often short, but short traces are not always concise, therefore, the reward targets conciseness directly, and a shorter length appears only as a side effect. as illustrated in figure 2 , all three examples use the same number of tokens, yet the first is the most concise, receiving the highest reward. the second and third are progressively less concise, resulting in lower rewards. unlike static length penalties, our llm-based conciseness score is dynamic and context-aware, enabling better generalization across problem types and difficulty levels while improving explainability. additionally, our method can be combined with correctness-based rewards, offering a flexible trade-off between conciseness and robustness. 4 . our main contributions are as follows: on the hardest questions, it outperforms full reasoning by 7.5% accuracy with 3.6× fewer tokens. on theoremqa, our method achieves 2.2% higher accuracy using 12.5× fewer tokens than the full reasoning model. these results highlight a level of length-efficiency control not achieved by any static heuristic or prior method. ## related work chain-of-thought (cot) @xcite enhances llm reasoning by prompting explicit intermediate steps, improving both performance and interpretability in complex tasks @xcite . variants such as tree-of-thought @xcite and graph-of-thought @xcite the local pond is partially frozen, and temperatures have dropped below freezing. should we provide food for the ducks? pond surface frozen, insects scarce; to prevent energy loss, we scatter 200 g cracked corn for the ducks. given assorted environmental circumstances, we might decide giving local ducks some food could help, like 200 g of cracked corn. when evaluating whether supplemental feeding is warranted, we review recent barometric readings, frost indices, daylight duration, and anecdotal observations of quacking frequency. field guides debate natural foraging versus human intervention, citing ecological balance and dependency risks. yet with pond ice thickening and caloric demands rising, distributing roughly 200 grams of cracked corn appears judicious. like deepseek-r1 (deepseek-ai et al., 2025; shao et al., 2024), combine supervised fine-tuning with rl. despite their performance, these methods are prone to generating verbose reasoning traces, a phenomenon known as the overthinking problem (chen et al., 2025; team et al., 2025). overthinking occurs when a model continues generating redundant reasoning steps after reaching the correct answer. this behavior wastes inference compute and risks introducing inconsistencies and logic errors @xcite . this issue is amplified by training objectives that reward long reasoning sequences @xcite , creating a misalignment between performance and efficiency. to address these issues, several approaches have been proposed for cot compression and length control. these include both reward-shaping strategies and optimization objectives designed to encourage shortness without compromising accuracy. methods like o1-pruner (luo et al., 2025a) use proximal policy optimization (ppo) @xcite with a length-harmonizing reward that penalizes deviations from a target length relative to a reference model. another work @xcite applies a two-branch ppo strategy with cosine penalties based on the prediction's proximity to a maximum acceptable length. l1 @xcite , whose aim is an exact token budget, impractical when problem difficulty is unknown, employs grouped reinforcement policy optimization (grpo) @xcite , explicitly conditioning the model to "think for n tokens" and penalizing outputs exceeding learned optimal lengths. simple preference optimization (simpo)based (meng et al., 2024) approaches like dast @xcite use data constructed from user or synthetic feedback to learn token-length preferences, bypassing reliance on reference models. other traditional @xcite and policy gradient (pg) methods (arora and zanette, 2025) have also been tried in this context, such as those including early fine-tuning with correctness-weighted penalties on output length. output length control, while previously a peripheral concern, has become central in reasoning llms @xcite @xcite . earlier works used architectural tweaks or static training signals, which lacked the flexibility to handle diverse inference scenarios. however, recent rl-based models @xcite @xcite achieve better control by continuously adapting the reasoning length based on the learned reward signals. yet many still rely on token count or heuristics as proxies for conciseness @xcite . models such as c3ot @xcite use llms as compressors to post-process reasoning traces, while continuous token methods like ccot @xcite and latent-space approaches like coconut @xcite reduce token-level redundancy by replacing explicit steps abstract representations. although effective at shortening outputs, these approaches operate post hoc or rely on static heuristics that limit adaptability across tasks. to address these challenges, we propose a novel approach that uses an llm as a judge to evaluate the conciseness of reasoning traces. while llms have been widely used as evaluators for factuality, summarization, coherence, and style @xcite @xcite , leveraging them to assess reasoning conciseness remains unexplored. our method introduces a semantically aware, dynamic reward function that goes beyond token count or static heuristics. this enables models to generate reasoning that is not only correct but also efficient and interpretable, bridging the gap between performance and inference cost. ## method we build on pre-trained multi-step reasoning models, improving them to generate correct answers using the minimum semantic effort through more concise outputs. instead of penalizing length, we reward conciseness: the ability to justify an answer with the fewest necessary and non-redundant steps. concise traces are often short, but short traces are not always concise, therefore the reward targets conciseness directly, and a shorter length appears only as a side effect. ## conclusion in this work, we introduce a novel, hyperparameterfree conciseness score used as a reward signal within any reinforcement learning framework and evaluated by an llm judge to encourage models to generate correct and concise reasoning traces. our approach improves semantic density without sacrificing accuracy, yielding strong efficiency-accuracy trade-offs. at equal token length, our traces are more informative, interpretable, and explainable than those from static length-penalized baselines due to fewer filler steps and tighter logic. overall, our results highlight the value of semantic rewards and llm-based judges for guiding concise and correct reasoning. in future work, we plan to evaluate our method across a wider range of model sizes to better understand how conciseness rewards scale. additionally, while we hypothesize that using more advanced judges like gpt-4o or gpt-4.5 could further enhance performance, we could not evaluate this due to our budget limitations and api-associated costs. we are also interested in combining our conciseness optimization with orthogonal methods, such as structure-aware search or explicit length conditioning, to enhance reasoning quality and efficiency. unlike other methods, ours is orthogonal and integrates well with approaches like l1 @xcite