Cautious Next Token Prediction (2025)

Abstract: Next token prediction paradigm has been prevailing for autoregressive models in the era of LLMs. The current default sampling choice for popular LLMs is temperature scaling together with nucleus sampling to balance diversity and coherence. Nevertheless, such approach leads to inferior performance in various NLP tasks when the model is not certain about testing questions. To this end, we propose a brand new training-free decoding strategy, dubbed as Cautious Next Token Prediction (CNTP). In the decoding process, if the model has comparatively high prediction entropy at a certain step, we sample multiple trials starting from the step independently and stop when encountering any punctuation. Then we select the trial with the lowest perplexity score viewed as the most probable and reliable trial path given the model’s capacity. The trial number is negatively correlated with the prediction confidence, i.e., the less confident the model is, the more trials it should sample. This is consistent with human beings’ behaviour: when feeling uncertain or unconfident, one tends to think more creatively, exploring multiple thinking paths, to cautiously select the path one feels most confident about. Extensive experiments on both LLMs and MLLMs show that our proposed CNTP approach outperforms existing standard decoding strategies consistently by a clear margin. Moreover, the integration of CNTP with self consistency can further improve over vanilla self consistency. We believe our proposed CNTP has the potential to become one of the default choices for LLM decoding. Code is available at https://github.com/wyzjack/CNTP.

## introduction large language models (llms) have advanced the capabilities of natural language processing * work was done while yizhou wang was an intern at adobe. recent research aims to bolster the reliability and depth of llm outputs through chain-ofthought (cot) reasoning, self-consistency voting, and iterative self-refinement @xcite @xcite @xcite . these approaches have demonstrated impressive improvements on qa and reasoning benchmarks by explicitly sampling multiple solution paths. however, extensive multi-sample strategies often inflate computational cost, while purely single-sample methods risk hallucinations and local optimum traps. furthermore, a great number of self-correction methods rely on external feedback signals @xcite , which can be intractable to deploy at scale. therefore, a method that dynamically adapts its exploration only when the model is unconfident, and does so efficiently, remains highly desirable for real-world deployment and application. in this paper, we propose a cautious next token prediction (cntp) approach that selectively samples multiple candidate paths whenever the model's prediction entropy is high, then automatically chooses the path with the lowest perplexity. through conditioning the sampling depth on confidence, cntp focuses computational resources precisely where the model is most uncertain, leading to stronger results in both purely linguistic and multimodal tasks. extensive experiments show that cntp consistently outperforms common decoding approaches. in summary, we make the following contributions: llms have demonstrated emergent reasoning abilities through chain-of-thought (cot) prompting, en-abling them to surpass earlier approaches on multistep inference tasks @xcite . self-consistency @xcite ) extends cot by sampling multiple reasoning paths and selecting the most frequent solution, yielding significant gains on math and commonsense qa tasks.adaptive-consistency @xcite halts sampling early if partial agreement is reached. the tree-of-thought framework explores multiple candidate "thought" sequences in a search-like manner, substantially improving performance on puzzles and longer-horizon tasks @xcite . in multimodal contexts, recent works incorporate vision inputs into multi-stage reasoning, achieving state-of-the-art results on image-and video-based qa @xcite . compared with these methods, our approach offers a simpler yet more robust mechanism for stepwise reasoning without relying on extensive prompt engineering or heavy sampling, yielding more effective inference. llm test-time sampling at inference time, widely used stochastic strategies such as top-p sampling or top-p (nucleus) sampling @xcite balance diversity and fluency, becoming the default choices for modern llms particularly in industrial products @xcite @xcite . more recently, min-p sampling @xcite puts forward a dynamic truncation method which adjusts the sampling threshold contigent on the model's confidence by scaling based on the top token probability. as to deterministic sampling strategies, greedy decoding selects the single most probable token at each step by choosing the one with the maximum token probability. beam search @xcite keeps track of a fixed number of top candidate sequences at each step and finally selects the trial with the highest joint probability. unlike these strategies, our method adaptively samples multiple paths at high-entropy steps, using perplexity to select the best continuation, thereby enhancing answer quality without losing stochasticity. llm self-correction llms sometimes produce errors or hallucinations, prompting the development of self-refinement or self-correction. @xcite shows that llms can self-correct by taking advantage of external tools to validate their initial outputs, gathering feedback on specific aspects, and then refining the content based on that feedback iteratively. @xcite proposes a finetuning strategy to make llm self-correct by en-algorithm 1 cautious next token prediction ## conclusion we introduce cautious next token prediction (cntp), a novel training-free decoding approach for llms that focuses additional computation on highuncertainty steps. by sampling multiple candidate paths only when entropy is high and selecting the path with the lowest perplexity, cntp achieves consistent improvements on both unimodal (text-only) and multimodal tasks. our experiments demonstrate that cntp outperforms standard sampling techniques and requires less overhead than multisample methods like self-consistency. in the future, we plan to extend cntp on autoregressive models beyond text generation, such as image generation @xcite .