HARP : Hesitation-Aware Reframing in Transformer Inference Pass (2025)

Abstract: This paper aims to improve the performance of large language models by addressing the variable computational demands in inference steps, where some tokens require more computational resources than others. We present HARP, a simple modification to “off-the-shelf” Transformer forward pass. Drawing from hesitation and the framing effect in decision-making, HARP selectively applies additional computation when the model encounters uncertainty during token generation. Our method mimics human cognitive processes by pausing at difficult decision points and reframing inputs for a different perspective. Unlike other approaches, HARP is model-agnostic, training-free, and easy to implement. We evaluate our method across various downstream tasks and model sizes, demonstrating performance improvements up to +5.16%. Notably, HARP achieves these gains while maintaining inference times twice faster than beam search. Simple and yet with significant gains, HARP provides insights into the potential of adaptive computation for enhancing the performance of Transformer-based language models.

## introduction causal language models based on the transformer architecture @xcite use a constant number of layer traversals to generate each new token. while this architecture is beneficial to provide easy parallelization during training @xcite , it may not fully leverage the model's full potential during inference, where tokens are generated sequentially. research in adaptive computation (e.g. @xcite @xcite suggests that inference steps are not equally challenging, with some being "harder" and others "easier." intuitively, these more challenging tokens would benefit from additional computational resources * corresponding author. to improve accuracy. unfortunately, the current transformer architecture treats each token equallyregardless of its difficulty-potentially leading to imprecision and performance drops. to address this limitation, a trend in language models has been to simply scale up models in size @xcite , allowing for more computation. while the difficult tokens benefit from this scaling, it also leads to unnecessary overhead for easier tokens. to tackle this uniform additional computation, speculative decoding @xcite employs a bigger model to verify and correct the tokens generated by the smaller model, acting as an external verifier. considering the smaller model as the original model, the larger model performs additional computations to ensure the quality of the generated tokens. this enables the system to spend more computational resources on complex tokens while preserving efficiency for easier ones. however, this approach requires the involvement of an external model. similarly, @xcite add fixed pauses during inference, using "pause tokens," to allow for additional computation on harder to-kens. while their method improves the generation of harder tokens, it requires full training and finetuning, and it still applies uniform additional computation when simpler tokens do not need the extra time. we draw inspiration from human behaviors to allow models to perform additional computations for "harder" steps without relying on external models or requiring retraining. two cognitive effects stand out: the (1) hesitation and the (2) framing effect. first, hesitation reflects uncertainty in decision-making. humans tend to pause and reconsider when faced with difficult decisions @xcite such that more effort is spent on complex inputs-aligning with the idea of "harder" tokens during inference. second, the framing effect indicates that how information is presented can influence judgment and response @xcite . it implies that a different representation of the same input can lead to better outcomes. in the following, we will refer to this another-view representation as "reframing" the inputs. building on these human-inspired concepts, we introduce hesitation-aware reframed forward pass (harp), a plug-and-play modification to the transformer's forward pass. to illustrate, we summarize harp on the right side of figure 1 . we begin with the standard forward pass, processing the inputs through the embedding layer and subsequent layers to produce initial logits. we then evaluate the hesitation of the model by computing its uncertainty over the logits (detailed in section 3.2). if the model is not hesitating, we directly output the initial logits. however, when the model is in a hesitation state, we reframe the inputs to infuse a different representation (explained in section 3.3). to do so, we perturb the embeddings and perform a new forward pass. finally, we combine the logits from the original and the additional forward pass, producing a final output incorporating both perspectives. by selectively reframing inputs, harp mimics human reconsideration under uncertainty, achieving up to 5.16% performance improvements with minimal additional cost. our method outperforms beam search @xcite across multiple tasks, offering higher accuracy and significantly faster inference. the code 1 is publicly available. our contributions can be summarized as follows: ## methods in this section, we present our hesitation-aware reframed forward pass (harp) method. we aim to perform an additional forward step from a different perspective when the model encounters uncertainty. we begin by reviewing the standard forward pass of transformers. then, we introduce the two key components of harp: quantifying uncertainty during inference and reframing inputs. finally, we will integrate these components to present the complete algorithm. ## results harp improves the performance of all tasks. in summary, our harp allows further improvements as high as +5.16% of high-end models that already show strong performance on many tasks without requiring any fine-tuning. this demonstrates the potential of harp to enhance state-of-the-art models. harp is model-agnostic. we show that harp also consistently improves models ranging from 3 to 8 billion parameters, including llama-3.1, mistral v0.3, and phi 3.5 mini. it illustrates its robustness and wide applicability across diverse language models. ## analysis uncertainty guides the additional computations. to investigate the effectiveness of our uncertainty-based approach (denoted as 'w\ shannon'), we compare it with a method that unconditionally adds an extra forward step for every token (denoted as 'w\o shannon'). this comparison helps us understand whether the improvements come solely from the additional computation or from our uncertaintyselected approach. our analysis reveals that unconditionally adding an extra forward step is not universally beneficial. while it improves performance compared to the vanilla model in most cases, especially in tasks like gsm8k and lambada, it also results in lower or equal performance on tasks like multiple-choice questions (commonsenseqa and mmlu pro). the uncertainty-conditional method outperforms the unconditional approach, even though it is sometimes marginal. this suggests that uncertainty is an excellent signal for selecting the steps requiring extra computation. a key advantage of the selective approach is its computational efficiency. as shown in the cost column of while additional computation can generally improve performance, our uncertainty-based approach offers a more nuanced and efficient solution. it achieves comparable or superior results to unconditional computation while balancing performance gains and computational costs. ## conclusion this paper presented a novel method, harp, designed to enhance language model inference without requiring retraining or architectural modifications. our approach uses a selection process based on token-level uncertainty to identify when additional computation is advantageous and an innovative method for reframing inputs during inference. we demonstrated that harp can achieve significant performance improvements across various tasks and model sizes, with accuracy gains of up to 5.16%. importantly, harp maintains inference efficiency compared to methods like beam search, with only an x1.25 average increase in inference time over the standard model. while harp provides a promising proof of concept, its real-world application is currently limited by challenges such as cache invalidity and the introduced randomness.