Think Clearly: Improving Reasoning via Redundant Token Pruning (2025)

Abstract: Recent large language models have shown promising capabilities in long-form reasoning, following structured chains of thought before arriving at a final answer. However, we observe that these reasoning paths tend to include substantial redundancy; analyzing attention patterns reveals that attention scores are widely scattered, particularly incorrect answers exhibit greater attention sparsity. In this paper, we demonstrate that deliberately removing this redundancy in the reasoning process significantly improves the performance through clear thinking (i.e., removing distraction). Specifically, we systematically identify such redundancy by measuring token-level attention scores to a special end-of-thinking token, which is appended to an explicit instruction inserted to conclude each intermediate reasoning step. Furthermore, we propose structure-aware pruning that prioritizes removing tokens in low-contributing reasoning chunks over individual tokens. After evicting redundant tokens, we remove the injected end-of-thinking instruction, then resume the reasoning generation. We demonstrate that our method significantly improves the over all accuracy across reasoning-intensive benchmarks without any training involved. In particular, our method shows strong performance on challenging mathematics competition benchmarks such as AIME and AMC, where reasoning redundancy is more prevalent.

## introduction large language models (llms) have demonstrated remarkable progress in complex reasoning tasks @xcite , including mathematical problem solving @xcite , multi-hop question answering @xcite , and long-form instruction following @xcite . this success is often attributed to † work done during an internship at amazon. ‡ work done at amazon. the emergence of structured reasoning chains, generating sequences of intermediate thoughts that gradually lead to a final answer @xcite . these reasoning chains allow models to break down complex problems into smaller, more manageable subproblems, mimicking the step-by-step cognitive strategies humans reasoning @xcite . in particular, recent reasoning models are trained to verbalize their internal thoughts, effectively leveraging the language abilities of pre-trained models @xcite . additionally, this verbalization offers a key advantage: it allows users to monitor and analyze-or even intervene in-the model's thought process during generation @xcite . in this paper, we found a somewhat interesting observation by monitoring this internal thought process of reasoning llms: reasoning chains often consist of significant redundancy. specifically, the model generates intermediate reasoning steps that tend to be repetitive, verbose, or include speculative detours that do not ultimately contribute to the final answer. such redundancies are also observed by analyzing the attention patterns, where attention distributions during reasoning typically consist of sparse patterns. this is especially problematic as llm can be easily distracted by irrelevant or redundant context @xcite , and this tendency is particularly evident when incorrect answers are generated (see fig. 1 ). more intriguingly, we observe reasoning chunks that receive consistently low attention from subsequent tokens, suggesting that the model briefly explores these misleading paths but eventually abandons them, leaving behind redundant traces in the generated sequence. this raises a key question: can we improve the performance by identifying and removing redundant tokens on-the-fly during the reasoning process? to this end, we propose a simple yet effective test-time token pruning method that removes redundant reasoning tokens. the core idea is to dynamically eliminate redundant tokens during generation, thereby enabling the model to preserve only the most critical reasoning steps necessary for reaching the correct answer. specifically, we propose two components: (i) identifying redundant tokens by measuring their contribution to a summarization-inducing end-of-thinking token, and (ii) structure-aware pruning that prioritizes removing low-contributing reasoning chunks rather than individual tokens. motivated by the observation that redundant tokens tend to receive low attention from the token that concludes the reasoning step, we inject an explicit instruction that prompts the model to summarize and terminate the current thought process, enabling redundancy measurement at intermediate stages. * rather than removing tokens in isolation, we first detect reasoning chunks that are unlikely to contribute to the final answer (i.e., misleading paths), and prune tokens within those chunks. once pruning is completed, we resume the generation by removing the injected end-of-thinking instruction. we conduct a comprehensive set of experiments to evaluate our token pruning scheme, focusing on reasoning-heavy scenarios across a wide range of tasks and models. our results demonstrate that this simple and lightweight inference-time approach can significantly improve reasoning performance. in particular, our method yields significant gains on challenging mathematics competition benchmarks such as aime and amc, where the model tends to generate more redundant reasoning steps, making our pruning approach especially effective. for instance, our method improves the original model's accuracy from 75.0% to 82.5% on amc2023 (ai-mo, 2023), while reducing kv cache memory usage by 10.3% on qwen2.5-7b @xcite . ## related work we provide a comprehensive review of related works on reasoning (focusing on the long chain-ofthought literature) and token pruning and compression frameworks. long chain-of-thought. recent large reasoning models (lrms) @xcite have increasingly adopted explicit chain-ofthought (cot) reasoning to enhance performance on complex tasks such as math @xcite , science @xcite , and symbolic reasoning @xcite . instead of producing direct answers, these models generate multi-step intermediate reasoning traces, allowing them to break down complex problems into smaller, more manageable steps @xcite . this long-form reasoning improves both accuracy and robustness @xcite , as it provides opportunities for self-correction (kumar et al., 2025), verification @xcite , and intermediate supervision during training or inference @xcite . a common pattern involves separating the reasoning phase from the answer phase, either through special tokens or by internal abstraction, where the reasoning is hidden and only summarized @xcite . some models expose the entire reasoning trace to the user to increase interpretability and transparency @xcite , while others keep it latent to reduce vulnerability to prompt manipulation or over-reliance @xcite . this shift toward structured, multi-step reasoning has been central to the recent progress of reasoning-focused llms, enabling strong generalization to diverse domains, from mathematics to program synthesis @xcite . in this paper, we propose an efficient yet effective test-time reasoning method by pruning redundant tokens, enabling the model to focus on critical points during long thinking. token pruning and compression. as the context length and generated sequences of large language models (llms) increase, the memory cost of self-attention becomes a significant bottleneck @xcite . in particular, the key-value (kv) cache, which stores past activations for each generated token, grows linearly with the sequence length. to address this, several existing works have explored strategies to identify and evict redundant tokens in kv cache @xcite . @xcite observe that attention distributions often exhibit strong focus on initial tokens, and show that retaining only the initial and most recent tokens is sufficient to preserve performance. other approaches leverage attention-based metrics to guide kv eviction @xcite . for example, @xcite proposes accumulating attention scores over decoding steps and using them as token importance indicators. oren et al. (2024) evict the token with the lowest attention score at each decoding step. @xcite observe that the number of crucial tokens varies across layers, motivating a layer-wise compression strategy. while these methods primarily aim to improve inference efficiency, our approach focuses on evicting redundant reasoning tokens to improve the performance (compared to the full kv cache). nevertheless, we demonstrate the potential of using our method as a reasoning compression scheme, achieving competitive performance under memory constraints and often exceeding recent token eviction and compression baselines. ## not all tokens matter for reasoning in this section, we investigate whether reasoning models truly require all previously generated tokens to reach a correct final answer. to this end, we focus on recent reasoning llms output t output , which consists of multi-step intermediate reasoning traces t reason followed by final answer t answer with special @xmath0... </think>. ## improving reasoning with redundant token pruning in this section, we propose a novel kv cache eviction policy that can improve reasoning models' effectiveness and efficiency by removing redundant tokens. specifically, we suggest a novel scoring function driven by self-summarization to identify redundant tokens (in section 4.1), and introduce a stepwise eviction policy that aggressively removes kv cache in the redundant reasoning step (in section 4.2). the overall procedure is illustrated in algorithm 1. ## experiments in this section, we present a thorough evaluation of our proposed framework, with the goal of verifying its ability to improve both reasoning accuracy and inference efficiency. our evaluation is divided into three parts: (1) effectiveness, which examines how well our method improves final answer correctness across a range of mathematical reasoning tasks ( we empirically demonstrate that our method not only reduces the computational overhead typically associated with long-form reasoning, but also improves accuracy by filtering out redundant and misleading intermediate steps. across all tested datasets and model sizes, our approach outperforms existing kv cache compression baselines, often by a significant margin. importantly, the gains are achieved without retraining or additional supervision, indicating that our method is broadly applicable as a plug-and-play enhancement for any autoregressive reasoning model. ## conclusion in this paper, we propose a test-time scaling method for enhancing reasoning in large language models by identifying and pruning redundant tokens during the generation of reasoning traces. by introducing a forced summarization phase at intermediate reasoning steps, we estimate the contribution of each token to the ongoing reasoning process. combined with a stepwise budget allocation strategy, our method demonstrates that targeted kv cache pruning can not only serve as a compression mechanism but also improve reasoning performance. moreover, our eviction algorithm is plug-and-play, making it suitable for memory-constrained settings, where it outperforms existing baselines under high compression scenarios.