“Well, Keep Thinking”: Enhancing LLM Reasoning with Adaptive Injection Decoding (2025)

Abstract: Large language models (LLMs) exhibit strong reasoning abilities, often attributed to few-shot or zero-shot Chain-of-Thought (CoT) prompting. While effective, these methods require labor-intensive prompt engineering, raising the question of whether reasoning can be induced without reliance on explicit prompts. In this work, we unlock the reasoning capabilities of LLMs without explicit prompting.Inspired by zero-shot CoT and CoT-decoding, we propose a novel decoding strategy that systematically nudges LLMs to continue reasoning, thereby preventing immature reasoning processes. Specifically, we monitor the model’s generation and inject a designated phrase, whenever the model is likely to halt or drift away from logical reasoning process. Our experimental evaluations on diverse reasoning benchmarks demonstrate that our proposed strategy substantially improves LLM reasoning capabilities, highlighting the potential of decoding-based interventions as an alternative to traditional prompting techniques.

## introduction large language models (llms) have revolutionized natural language processing (nlp), demonstrating remarkable performance across a wide range of tasks, including question-answering, code generation, and machine translation. scaling up model size has led to improvements in fluency, generalization, and sample efficiency @xcite . however, despite these advancements, llms continue to struggle with complex reasoning tasks such as arithmetic, commonsense reasoning, and multi-step logical inference. to address these challenges, researchers have primarily relied on prompt engineering techniques, such as fewshot chain-of-thought (cot) prompting and zeroshot @xcite @xcite figure 1 : overview of our proposed method, adaptive injection decoding, with injecting a designated phrase "well" whenever the model is likely to conclude its response immaturely with @xmath0. @xcite @xcite . these approaches guide llms by providing structured reasoning examples or explicit step-bystep instructions, significantly improving performance on various reasoning benchmarks. however, they exhibit key limitations in that few-shot prompting requires human-crafted exemplars, while zeroshot prompting is highly sensitive to prompt phrasing. more recently, some studies have explored cot reasoning without prompts by modifying the decoding strategy @xcite . additionally, a test-time scaling method has been introduced, which intervenes during inference to control test-time compute @xcite . despite various efforts to enhance llm reasoning, existing approaches have not directly addressed the fundamental causes of their reasoning failures. through our analysis, we categorize these failures into three primary types, highlighting the need for a more targeted intervention. first, models terminate generation without producing a response to given questions, returning only @xmath1, a phenomenon we refer to as silence. second, models often produce responses that lack actual logical reasoning, responding with uninformative outputs or irrelevant patterns, such as numeric sequences or question repetition, a failure type we call no reasoning. finally, in some cases, models initiate reasoning but halt prematurely, leading to incomplete or logically inconsistent answers, which we refer to as incomplete reasoning. to address these reasoning failures, we posed a fundamental question: how do humans think when solving complex problems? when solving complex problems, humans rarely produce an immediate answer. instead, they pause, reflect, and often restart their thought process before reaching a final conclusion. this process helps sustain reasoning and leads to more thoughtful and complete solutions. inspired by this, we propose adaptive injection decoding (aid), a novel decoding strategy that performs test-time intervention to improve the model's reasoning process. our approach is distinct from conventional decoding strategies in that it injects a simple guiding phrase during inference to support the model's reasoning. this intervention is triggered dynamically, only when the model shows signs of prematurely terminating or drifting away from logical reasoning. our contributions are as follows: ## observations: immature reasoning we conducted an experiment using llama 3.1-8b on the gsm8k dataset to analyze typical failure modes in its reasoning attempts. our analysis identifies recurring patterns in incorrect responses, offering insight into the model's limitations and potential areas for improvement. the following sections summarize key observations, with specific examples provided in appendix d. we identify immature reasoning as a major factor behind the poor reasoning performance of llms. immature reasoning refers to instances where the model deviates from or fails to sustain a coherent reasoning trajectory, leading to incomplete, irrelevant, or absent responses. our analysis on the model's responses revealed three distinct types of immature reasoning. ## adaptive injection decoding to mitigate immature reasoning, we introduce a novel decoding strategy. our approach, adaptive injection decoding (aid), inserts a predefined injection phrase (e.g., "well") as the next token dynamically whenever the probability of @xmath2. this method is designed to prevent the model from prematurely algorithm 1 adaptive injection decoding require: model f (•), prompt x, top-k k, injection prompt p, using zero-shot-cot zs_cot, maximum length max_length 1: if zs_cot then 2: given an input query x as an input prompt, the model f generates a sequence of response tokens r 1 , r 2 , . . . , r t through greedy decoding. at each decoding step t, if @xmath3, it is replaced with injection phrase p. the modified sequence r 1 , r 2 , . . . , r t , p is then fed back into the model as the next input. the algorithmic details of our method are provided in algorithm 1. to identify the most effective injection phrase, we explored a range of candidates, including single words, conjunctions, full sentences, and machine language expressions. among these, "well" consistently outperformed other alternatives, as shown in by analyzing token-level probability distributions, we found that different models exhibit distinct behaviors in handling @xmath4. for instance, llama-3.1-8b @xmath5, whereas mistral-7b-v0.3 and gemma-3-7b maintain it as a highprobability candidate before eventually emitting it later in the sequence. building on this observation, we adopted a strategy that triggers the injection @xmath6, instead of restricting it to cases where it holds the top rank. further investigated optimal top-k values for each model in subsequent experiments (see section 4.3 for details). ## related works a variety of methods have been explored to enhance the reasoning capabilities of llms. these techniques can be broadly categorized into tuningbased, prompt-based, sampling-based, and testtime approaches. below, we contextualize our ap ## conclusion in this work, we addressed the challenge of immature reasoning in llms, wherein the model's reasoning process terminates prematurely, leading to silence, no reasoning, or incomplete reasoning. to mitigate this, we proposed a novel decoding strategy. rather than heavily relying on prompt engineering, our approach injects a designated "nudging" phrase whenever the model is likely to produce @xmath7. this lightweight, testtime intervention encourages continued reasoning by acting as an implicit signal that mirrors human thought patterns. extensive experiments across diverse reasoning tasks-including arithmetic, commonsense, and logical reasoning-demonstrate that our method substantially improves performance in both zeroshot and zero-shot cot settings. moreover, it promotes reliable reasoning outcomes by compensating for weak prompts and adapting to varying levels of task complexity. overall, our proposed method, adaptive injection decoding (aid), offers a promising direction for unlocking the latent reasoning potential of llms while maintaining computational efficiency and ease of deployment. future work could explore dynamically selecting injected phrases based on task characteristics, extending the method to more complex generation tasks, or integrating it with advanced sampling strategies to further improve robustness and reliability.