Smarter, Not Harder: Training-Free Adaptive Computation for Transformers (2025)

Abstract: Adaptive Computation in Transformers (ACT) has been pursued in two directions: efficiency- and performance-focused. We study performance-focused ACT, or PACT, which invests more computation on hard steps to improve performance, such as by adding forward passes. We first discuss beam search and hesitation-based methods as PACT and their limitations. While the hesitation-based approach outperforms beam search by perturbing input embeddings, it suffers from inefficiency due to invalidating KVCache and exhibits instability due to its reliance on randomness. To address this, we propose IMPACT, a novel PACT method that perturbs network weights rather than input embeddings. This approach enables the reuse of KVCache, offers deterministic predictions, and significantly improves memory and computational efficiency. By achieving a better balance between performance and efficiency, IMPACT makes PACT accessible to communities with consumer-grade hardware.

## introduction scaling the size and pretraining of large language models (llms) has revolutionized their capabilities, enabling remarkable performance across a wide range of tasks @xcite . however, this scaling also makes retraining llms prohibitively expensive, motivating recent approaches for improving inference. one promising direction for this is adaptive computation in transformers (act), a paradigm that dynamically adjusts computation at the token level to handle varying complexities during inference. act has been explored along two primary directions: efficiency-focused and performancefocused approaches. the former aims to reduce computation for simpler inferences, while the latter seeks to enhance performance by allocating more computation to complex inference steps. beam search @xcite , a constrained variant of breadth-first search that explores multiple hypotheses at each step, is often adopted for enhancing performance at the cost of increased inference time for enumerating multiple hypotheses. while originally designed to mitigate the label bias problem in greedy decoding, rather than for such performance-focused act (pact) uses, its effectiveness as a pact has been explained through the uniform information density (uid) hypothesis @xcite . according to uid, human-generated sentences tend to distribute information evenly, and beam search aids in making the surprisal values of model outputs more uniform, resulting in more human-like text. a stronger example of pact is hesitation-based methods, such as harp @xcite , as more directly aligned with uid. harp introduces 'hesitation' into high surprisal token generation, mimicking how humans deliberate on complex decisions. specifically, it first identifies points of hesitation and then performs additional forward passes to generate reframed predictions, which can be implemented using random dropout on input embeddings @xcite . despite their potential, existing hesitation-based methods face significant drawbacks, including increased latency and memory overhead. by randomly perturbing input embeddings, these methods invalidate key-value caches (kvcache), leading to inefficiencies. furthermore, randomness can sometimes make the performance of harp inferior. to address these challenges, we propose im-pact, a pact method designed to better balance performance and efficiency. first, impact enables kvcache reuse, critical for complex tasks requiring a long generation, a significant demand for recent reasoning-focused models. impact builds on the principles of harp but replaces the reframing through input embedding dropout with an approximation of the original network. second, by eliminating random perturbations, im-pact ensures a fully deterministic forward pass, improving both reliability and efficiency. unlike harp, which often requires multiple kvcacheinvalidated runs due to its stochastic nature, we show comparable gains without compromising generalization. we empirically validate impact on reasoningintensive tasks and modern approaches to reasoning, such as chain-of-thought. qualitatively, we demonstrate that hesitation-based methods, such as impact, align closely with the uid hypothesis. additionally, impact is effective across models of various scales-from llama-3.1-8b-instruct to llama-3.3-70b-instruct. the code for this method is publicly available. foot_0 ## preliminary and related work this section overviews token-level pact approaches, though other pact approaches such as pause tokens @xcite , or test-time scaling @xcite , can be applied in conjunction for further improvements. ## impact: efficiency improved pact this section discusses the inefficiencies associated with uid computation and introduces impact as a solution. ## experiments we use llama-3.1-8b-instruct and llama-3.3-70b-instruct @xcite as our base models. all evaluations are conducted using a batch size of 1, in a greedy decoding setting (temperature of 0). latency is evaluated on the whole dataset using a batch size of 1 and a concurrency of 1, following the convention of harp. for beam search, we use a beam size b of 3, and length-normalization @xcite of 0.6, to be consistent with previous work @xcite . we detail the experimental settings in appendix e. impact vs. other pact methods: balancing performance and efficiency performance on complex reasoning tasks chain-of-thought prompting @xcite encourages the model to generate long reasoning steps, often prone to hesitation, making the generation process more complex. ## conclusion to address the limitations of the hesitation-based pact method, we introduced a cache-aware alternative that perturbs network weights instead of the inputs. our method, impact, preserves the training-free advantage of harp while improving efficiency and ensuring determinism by eliminating the randomness caused by dropout. we validated improvements in both performance and efficiency, complementing existing approaches focused on performance and efficiency. this makes higher performance and computational speed accessible to real-world applications and to a wider audience, including those with consumer-grade gpus and limited memory.