A Study on Leveraging Search and Self-Feedback for Agent Reasoning (2025)

Abstract: Recent works have demonstrated that incorporating search during inference can significantly improve reasoning capabilities of language agents. Some approaches may make use of the ground truth or rely on model’s own generated feedback. The search algorithm uses this feedback to then produce values that will update its criterion for exploring and exploiting various reasoning paths. In this study, we investigate how search and model’s self-feedback can be leveraged for reasoning tasks. First, we explore differences in ground-truth feedback and self-feedback during search for math reasoning. Second, we observe limitations in applying search techniques to more complex tasks like tool-calling and design domain-specific approaches to address these gaps. Our experiments reveal challenges related to generalization when solely relying on self-feedback during search. For search to work effectively, either access to the ground-truth is needed or feedback mechanisms need to be carefully designed for the specific task.

## introduction search algorithms have traditionally relied on ground-truth feedback, particularly in domains like games where win/loss conditions provide clear signals for decision-making @xcite . some recent works have shown potential in incorporating search during model inference to improve reasoning @xcite @xcite . typically, the search process requires feedback on the correctness of the candidate solutions. while such feedback traditionally came from external verification, these recent works have explored using the model's own assessment as feedback during search. the use of self-feedback is motivated by emerging evidence of agent's capability for selfcorrection and self-refinement @xcite * work done as an intern at aws @xcite . this direction is particularly exciting as it suggests the potential for models to guide their own search process and evolve without relying on external verification, moving us closer to more generalized, autonomous agents @xcite . however, when implementing search for reasoning, a critical question emerges: how valuable is the model's own feedback in guiding the search process? while models may be capable judges in some scenarios @xcite , their ability to reliably assess their own outputs during search remains questionable. this becomes particularly important when ground-truth feedback is unavailable, as is often the case in real-world applications. in this study, we investigate this question through two distinct tasks: mathematical reasoning, gsm8k @xcite , and tool-calling, tooltalk @xcite . our analysis on gsm8k reveals that while search itself is valuable, using the model's self-feedback to guide the search may not be optimal. instead, other strategies, like majority voting across multiple nodes during search, proves to be more effective. we then extend our investigation to tool-calling, a more complex domain involving strategic decisionmaking and parameter selection. here, we find that search with self-feedback can actually degrade performance. this leads us to explore domain-specific approaches for augmenting feedback mechanisms, including in-content examples and specialized verification modules. while the promise of self-improving models through feedback and search remains appealing, the current reality of using self-feedback in search needs to be carefully reconsidered. this highlights a gap between models' previously demonstrated self-correction capabilities in some scenarios and their reliability in guiding search processes for reasoning. overall, this suggests the need for more engineered feedback mechanisms tailored to spe-cific tasks @xcite or alternative approaches for leveraging self-feedback outside of search @xcite . these findings are aligned with recent generative ai breakthroughs that also mention the limitations of search and selffeedback for agent reasoning @xcite . ## related work recent research has extensively explored methods to enhance llm performance through advanced test-time compute approaches. @xcite investigates scaling behaviors of sampling strategies, while @xcite show positive results with a simple test-time scaling approach called budget forcing. recent work highlight emerging potential for self-feedback @xcite . likewise, there is also critique on whether models can truly evaluate and correct their own outputs @xcite . beyond sampling and iterative refinement, other works propose using monte carlo tree search (mcts) as a more structured approach to improve reasoning @xcite @xcite . mcts can explore the search space effectively, trading off exploration and exploitation using the u ct criterion: ## search without ground-truth feedback in this section, we explore the use of the model's own feedback for search on math reasoning dataset gsm8k. we build on the mctsr (mcts with self-refine) framework @xcite , which computes q value based on the model's feedback of its generated solution. one caveat with mctsr is that the method originally assumes access to ground-truth feedback for early stopping and answer selection. access to ground-truth feedback is appropriate for games like chess and go. in other scenarios, there may be a lack of access to ground-truth feedback. this raises the question: can we still leverage the model's self-feedback or other signals to guide search effectively? to address this, we investigate alternative strategies for answer selection and evaluate their performance in the absence of ground-truth feedback. ## search with augmented feedback this section explores the application of search to a more complex domain of tool-calling, specifically using the tooltalk dataset @xcite . unlike math reasoning tasks where there is one answer and verification is relatively straightforward, tooltalk presents a significantly more nuanced challenge. the dataset consists of multiturn dialogues where agents must understand user intents, decide when to make tool calls versus asking for clarification, and ensure all tool parameters are grounded in the conversation context. tooltalk evaluation is done using teacher forcing, where we condition the conversation history based on the ground-truth conversation and evaluate the agent's response. within each turn, there can be zero to multiple tool calls, and tooltalk evaluation sequentially decodes one tool call at a time. the complexity of tooltalk stems from its openended nature and the strategic decisions required at each turn. agents must not only understand what tools are available but also determine the appropriate moment to use them. a successful response often requires maintaining coherence across multiple turns while avoiding a common pitfall: parameter hallucination, where models fabricate plausible but incorrect tool parameters. this represents a fundamental departure from math reasoning tasks, where the challenge lies primarily in computational logic rather than strategic decision-making. ## conclusion in this work, we conduct a study on integrating selffeedback into search for agent reasoning. while search remains a valuable technique for enhancing model performance, our results demonstrate that relying on self-feedback may be suboptimal or even detrimental in certain contexts. these insights have implications for the development of generalized, autonomous agents. rather than pursuing purely self-guided approaches, our work indicates that successful search implementations may require carefully designed, domain-specific feedback mechanisms or hybrid approaches that combine self-refinement with other verification strategies.