SMART : Self-Aware Agent for Tool Overuse Mitigation (2025)

Abstract: Current Large Language Model (LLM) agents demonstrate strong reasoning and tool use capabilities, but often lack self-awareness, failing to balance these approaches effectively. This imbalance leads to **Tool Overuse**, where models unnecessarily rely on external tools for tasks solvable with parametric knowledge, increasing computational overhead. Inspired by human metacognition, we introduce **SMART** (Strategic Model-Aware Reasoning with Tools), a paradigm that enhances an agent’s self-awareness to optimize task handling and reduce tool overuse. To support this paradigm, we introduce **SMART-ER**, a dataset spanning three domains, where reasoning alternates between parametric knowledge and tool-dependent steps, with each step enriched by rationales explaining when tools are necessary. Through supervised training, we develop **SMARTAgent**, a family of models that dynamically balance parametric knowledge and tool use. Evaluations show that SMARTAgent reduces tool use by 24% while improving performance by over 37%, enabling 7B-scale models to match its 70B counterpart and GPT-4. Additionally, SMARTAgent generalizes to out-of-distribution test data like GSM8K and MINTQA, maintaining accuracy with just one-fifth the tool calls. These highlight the potential of strategic tool use to enhance reasoning, mitigate overuse, and bridge the gap between model size and performance, advancing intelligent and resource-efficient agent designs.

## introduction recent advancements in large language models (llms) @xcite @xcite have led to remarkable improvements in reasoning capabilities, driving progress in diverse domains such as coherent text composition @xcite , code generation @xcite @xcite , complex logical deduction @xcite , and nuanced natural language understanding @xcite @xcite . however, challenges remain, such as the inability to handle real-time information @xcite , model real-world challenges @xcite , provide accurate mathematical results @xcite , and fully comprehend human intentions @xcite . these limitations highlight the need for llms to leverage external tools @xcite @xcite , enabling them to function as agents capable of assisting users in diverse tasks @xcite . effective tool use and reasoning are thus complementary, each enhancing the other to overcome current shortcomings. therefore, in problem-solving, a language agent often combines reasoning with tool use, following a react-style approach @xcite , where the model alternates between thought processes and actions to derive solutions. this enables the core agent to apply its parametric knowledge to advance task-solving while using external tools to address its limitations. however, this interplay raises a critical question: when should the agent rely on external tools versus its own knowledge? to investigate this, we first conduct a preliminary study on both llms and lm-driven agent systems to assess their ability to dynamically and effectively switch between external tool use and parametric knowledge-driven reasoning. our empirical results reveal a consistent bias, with llms unnecessarily invoking tools over 30% of the time, and agent systems exhibiting similar behavior even when their parametric knowledge alone would suffice. we identify this phenomenon as tool overuse, which arises from the model's inability to recognize when its internal knowledge is sufficient. this not only leads to unnecessary resource consumption but can also confuse the model, ultimately degrading performance. this observation highlights the need for better calibration of an agent's self-awareness, ensuring it can discern when to rely on tools versus its own knowledge. striking this balance is crucial for enhancing efficiency, scalability, and user experience as lm-driven agents are increasingly deployed in real-world applications. to address this challenge, we propose smart (strategic model-aware reasoning with tools), which draws inspiration from human decisionmaking to calibrate self-awareness in agent models for effective tool use and reasoning. in metacognitive theory @xcite , psychology highlights humans' awareness of their thought processes, including when to apply specific problem-solving strategies @xcite . as figure 1 illustrates, this implicit heuristic allows dynamic balancing between external strategies and internal knowledge @xcite . similarly, agents need metacognition to optimize tool usage. by aligning the model's subjective perception with its knowledge boundary, we enable agents to make more informed decisions on when to rely on external tools or internal knowledge. we adopt a data-driven approach to calibrate model decision-making by constructing smart-er (smart-enhanced reasoning), a dataset spanning three domains-math, time, and intention. it addresses key llm limitations, including computational accuracy @xcite , outdated knowledge @xcite , and user preference awareness @xcite . specifically, each question in smart-er combines sub-questions the model handles well (e.g., simple arithmetic, static facts, commonsense) with those it struggles with (e.g., complex math, dynamic facts, userspecific intentions). we break down each question into reasoning steps, categorizing them as either parametric knowledge-driven or tool-dependent. for parametric steps, we provide reasoning based on internal knowledge. for tool-dependent steps, we map them to appropriate tools, execute them, and integrate the results into the reasoning process. finally, inspired by metacognitive heuristics, we refine each step with explicit justifications, clarifying when parametric knowledge suffices or external tools are needed. by transforming implicit decision-making heuristics into explicit languagebased reasoning, we guide the model to develop calibrated awareness of its knowledge boundaries. leveraging smart-er, we develop smartagent, a family of agent models designed to dynamically balance reasoning between parametric knowledge and external tools. empirical results show that smartagent reduces tool use by 24% while improving overall performance by over 37%, effectively mitigating tool overuse. notably, it enables 7b-scale models to match the performance of gpt-4 and 70b models, bridging the gap between model size and capability. additionally, smartagent efficiently handles out-of-distribution (ood) tasks, requiring only one-fifth the number of tool calls while preserving accuracy. finally, analysis of smartagent's confidence through logits reveals more certain reasoning-tool-switching decisions, further validating our approach in calibrating the agent's self-awareness. in summary: • we identify and define the issue of tool overuse, emphasizing that strategically balancing the complementary strengths of knowledge-driven reasoning and external tool calls can mitigate this problem in both llms and agent systems. ## preliminaries to investigate how models decide between invoking tools and relying on their own knowledge, we conduct a preliminary study on both llms and lm-driven agent systems. our findings reveal both llms and agent systems' strong tendency for excessive tool use, which we define as tool overuse, leading to unnecessary resource overhead. definition of tool overuse. tool overuse refers to the excessive reliance on external tools when an agent model could have successfully completed the task using its parametric knowledge alone. formally, let q be the total set of questions, and let p be the subset of questions that the model can correctly answer without using any tools. the model's intrinsic reasoning capability is then given by α = |p | |q| . now, suppose that when provided with access to tools, the model chooses to invoke at least one tool on a fraction β of these questions in p . the tool overuse rate is then defined as: @xmath0, highlighting inefficiencies in the model's decision-making process. experiments on llms. we first experiment with llama-3.1-8b @xcite and mistral-7b (jiang et al., 2023) on the gsm8k test set @xcite . each test question is presented under two conditions: i) the model reasons through the question normally and provides a final answer without using tools, and ii) the model has access to tools and independently decides whether to use them (see appendix a.2). the statistics in figure 2 reveal two key insights. first, both models exhibit significant tool overuse, with llama's rate exceeding 50%. second, in some cases, tool use leads to incorrect answers, even for questions the model could have solved correctly without external assistance. this highlights how excessive reliance on tools can introduce unnecessary complexity and degrade performance. experiments on lm-driven agents. in addition to llms, we also experiment with two agent systems: xagent (team, 2023) and agentgpt @xcite , both designed for complex problem-solving and driven by closed-source gpt models. we sampled 50 queries from the gsm8k test set that can be answered correctly without tools (see appendix a.1) and instructed the models to use tools only when necessary. the results in to address the challenge of tool overuse, we draw inspiration from how humans balance internal knowledge and external tools. metacognitive theory @xcite suggests that human decision-making relies on an implicit awareness of knowledge boundaries, enabling strategic, step-by-step problem-solving @xcite . inspired by this, we aim to equip agent models with a similar capability-calibrating their metacognition to optimize reasoning and tool use. to address this, we propose smart, a datadriven approach that enhances self-awareness in agent models. while llms acquire broad knowledge from large-scale corpora @xcite , they are not explicitly trained to recognize their own strengths and limitations. to bridge this gap, we introduce smart-er, the first dataset contrasting areas where models excel versus struggle. covering three domains with 3k+ questions and structured reasoning chains, smart-er helps agents strategically decide when to rely on internal knowledge or external tools. ## experiment in this section, we present results demonstrating smartagent's effectiveness in reducing tool overuse while enhancing reasoning performance. ## discussions agent's improper tool usage. our empirical analysis reveals a notable phenomenon of tool overuse, where agents frequently rely on external tools even when internal knowledge is sufficient. this over-reliance likely arises from two factors: i) the agent's uncertainty about its own capabilities, and ii) the perceived ease of external lookups compared to internal reasoning. we also observe instances of tool underuse, especially in large-scale models like gpt-4o and llama-70b, where agents neglect to call essential tools, possibly due to misjudging the complexity of the task. both overuse and underuse contribute to concerns over computational efficiency and solution accuracy. future research could explore methods to better balance these trade-offs, such as by introducing explicit resource constraints or budgets for tool calls. ## conclusion inspired by human metacognition in decisionmaking, we propose the smart paradigm for agent reasoning, where agents recognize their knowledge boundaries to decide when to use tools or parametric knowledge. specifically, smart-er refines this decision boundary by incorporating questions that highlight areas where current lms excel and struggle. using these curated reasoning chains, we train smartagent to better balance tool use and parametric knowledge, reducing tool overuse. our results show that a simple data-driven approach can effectively calibrate model awareness, paving the way for efficient, low-resource agent development where "smartness" stems from both performance and metacognitive ability to optimize the reasoning strategy.