Entropy-based Exploration Conduction for Multi-step Reasoning (2025)

Abstract: Multi-step processes via large language models (LLMs) have proven effective for solving complex reasoning tasks. However, the depth of exploration of the reasoning procedure can significantly affect the task performance. Existing methods to automatically decide the depth often lead to high cost and a lack of flexibility. To address these issues, we propose Entropy-based Exploration Depth Conduction (Entro-duction), a novel method that dynamically adjusts the exploration depth during multi-step reasoning by monitoring LLM’s output entropy and variance entropy. We employ these two features to capture the model’s uncertainty of the current step and the fluctuation of uncertainty across consecutive reasoning steps. Based on the observed entropy changes, the LLM selects whether to deepen, expand, or stop exploration according to the probability, which facilitates the trade-off between the reasoning accuracy and exploration effectiveness. Experimental results across four benchmark datasets demonstrate the efficacy of Entro-duction.

## introduction large language models (llms) have demonstrated remarkable reasoning capabilities across various domains @xcite @xcite @xcite . however, they would still encounter challenges in generating accurate and effective multi-step reasoning in terms of complex downstream tasks. specifically, llms may exhibit overreasoning or under-reasoning, which both imply the depth of exploration for a given problem does not match expectations @xcite @xcite . this mismatch issue of reasoning path could lead to several issues: (1) inaccurate, insufficient, or redundant reasoning outcomes; (2) unnecessary computation costs @xcite @xcite . these issues might be attributed to two aspects: i) the lack of evaluation and regulatory mechanisms for llms' reasoning process; ii) there are significant variations in the reasoning process required for different tasks, and llms might not be unable to accurately judge and adjust the depth of exploration for a task solely based on their parametric knowledge during pretraining. existing methods for optimizing the exploration depth of multi-step reasoning in llms can be categorized into two types: outcome-based optimiza-tion and process-based optimization. outcomebased optimization aligns the llms' reasoning exploration with human expectation after generating a complete reasoning path and approaching the final conclusions @xcite @xcite . these approaches rely on post-training techniques of llms, which are resource-intensive and do not provide an immediate improvement on the current task. further, due to the diversity of reasoning tasks, the enhancement gain is task-specific. in process-based optimization, the llms supervise and evaluate their reasoning process through self-critique or by labeling reasoning steps to enhance outputs @xcite @xcite . the advantages of the process-based optimization methods are their immediacy and low cost. however, the llms should have high reasoning capabilities and a substantial knowledge base. moreover, since the process is usually opaque, it is difficult for humans to effectively provide supervision signals and interpret the optimization processes. the model's illusions, biases, and errors may be reinforced during this process @xcite @xcite . our target. given the existing challenges, we aim to develop a method that guides llms to automatically, concisely, and transparently determine the appropriate depth of exploration based on task information and the model's reasoning state. the goal is to enable the model to look ahead during reasoning, plan dynamically, and decide whether further exploration is necessary to achieve optimal reasoning outcomes. the whole procedure involves enhancing multi-step reasoning performance and reducing unnecessary exploration. our method. to tackle these challenges, we propose entropy-based exploration depth conduction (entro-duction), a novel method to help llms assess the adequacy of exploration during multi-step reasoning processes, thus enhancing the outcomes of reasoning. inspired by entropy uncertainty measurement @xcite @xcite , we employ entropy changes in the llm's reasoning process to evaluate its uncertainty of reasoning, which reflects the reasoning confidence, and accordingly switch exploration strategy. specifically, we use entropy and variance entropy as rewards to update the llm's probability distribution for its next exploratory action, whether to deepen, stop, or expand exploration. this distribution subsequently guides a behavior selection mechanism that promotes reasoning when exploration is insufficient and avoids redundant reasoning when it is adequate. in summary, our contributions involve: ## experiments in this section, we first compare the reasoning performance and reasoning steps used between baseline methods and entro-duction. subsequently, we present ablation studies to analyze the contributions of each part of our strategies. following this, we examine how different parameter settings impact entro-duction's overall robustness. ## conclusion in this study, we introduce entro-duction, a novel approach that dynamically adjusts the exploration depth during llm multi-step reasoning by monitoring the entropy and variance entropy. entroduction leverages the change of both metrics to select exploration behavior to enhance reasoning performance and avoid redundant reasoning steps. our experiments across multiple reasoning datasets demonstrate the effectiveness of the entro-duction and its components.