SMARTCAL : An Approach to Self-Aware Tool-Use Evaluation and Calibration (2024)

Abstract: The tool-use ability of Large Language Models (LLMs) has a profound impact on a wide range of applications. However, LLMs’ self-awareness and self-control capability in appropriately using tools remains understudied. The problem is consequential as it alarms a potential risk of degraded performance and poses a threat to trustworthiness on the models. In this paper, we conduct a study on a family of state-of-the-art LLMs on three datasets with two mainstream tool-use frameworks. Our study reveals the tool-abuse behavior of LLMs, a tendency for models to misuse tools along with models’ frequent overconfidence in tool choice. We also find that this is a common issue regardless of model capability. Accordingly, we propose a novel framework, SMARTCAL, to mitigate the observed issues, and our results show an average 8.6 percent increase in the QA performance in three testing datasets and 21.6 percent lower Expected Calibration Error (ECE) than existing methods.

## introduction the tool-use ability of llms has a profound impact on a wide range of applications. agents that are fine-tuned on various human-computer interaction scenarios such as web browsing @xcite , code writing @xcite , or even internet shopping @xcite have been successfully deployed to streamline workflows and boost efficiency in multiple realms within the industry. recent research has also achieved impressive results by welding various tools into the step-wise reasoning of retrieval augmented generation (rag), such as a retriever @xcite , a database operator @xcite @xcite , or a collection of tools @xcite . while incorporating tools into llms is 1 our code and data are available at https://github.com/henrysyh2000/smartcal . critical for many applications, @xcite argue that the tool-use step can negatively impact the performance in some circumstances: e.g., when llms have reliable parametric memory. this motivates further studies exploring adaptive retrieval strategies @xcite . however, many existing tool-use frameworks rely on either passive in-context learning from existing few-shot examples @xcite @xcite or fine-tuning on dedicated datasets @xcite @xcite . the absence of a model's active thinking in tool-use thus leaves a crucial question under-studied: are llms aware of when to use which tool? to understand the performance of using tools, we conduct a series of experiments under the scenario of open domain qa @xcite . our results raise concerns related to the above question: the tracking of llm tool usage across chatgpt series (openai, 2023) and llama-3-instruct on entity questions data @xcite shows that on average, a model misuses one or more types of tools in over 20% of its total reasoning steps. additionally, when the model is asked to report its confidence in selecting a certain tool within each step, more than 90% of its stated confidence falls in the confidence bin where the reported confidence level is higher than the actual answering accuracy, indicating the model's overconfidence with respect to tool choice. the bottom part of the first two columns in figure 1 demonstrates such tool-abuse phenomenon. in this paper, we propose smartcal, a novel approach to helping mitigate tool-abuse. smartcal consists of three components (i) self-evaluation (se), (ii) confidence prior collection (cpc), and (iii) augmented reasoning (ar), which mitigate tool-misuse and provide a more reliable calibration performance. deployment of smartcal on two different tool-use frameworks, art @xcite and dsp @xcite , shows that it is able to derive an efficient strategy on tool-use and provides better calibrated answers. to the best of our knowledge, this is among the first efforts focused on investigating the calibration of llm-based tool-use. fostering proper use of tools is considered to be important for many applications that emphasize the alignment of llms @xcite . our contributions are summarized as follows: we observe tool-abuse in llms, which includes tool-misuse behavior and an inaccurate evaluation of verbalized confidence scores. we show that degradation in tool-use calibration remains a common issue regardless of increasing model capabilities. we introduce smartcal, a novel framework that aims to mitigate tool abuse. smartcal achieves an average of 8.6 percent increase in the qa performance and a 21.6 percent decrease in expected calibration error (ece) compared to baseline models. ## smartcal: a tool-use recalibration approach motivated by the self-verification feature that constitutes the reasoning capability in a multi-agent system @xcite , we introduce a novel framework smartcal that helps control tool-misuse based on multiple llm agents. different from existing approaches that emphasize incontext learning from demonstrations such as automatic multi-step reasoning and tool-use (art) @xcite shown in the left column in figure 1 and demonstrate search predict (dsp) @xcite , smartcal incorporates extra evaluation steps to examine the legitimacy of tool usage within each step. additionally, compared to existing tool-use frameworks where each step is controlled by a single agent, smartcal features an enhanced pipeline that promotes the collaboration among the agents, ensuring accurate and reliable tool usage during step-wise reasoning. specifically, when prompted with an input task, smartcal first derives an optimized strategy about when to use which tool. then, the collaboration between specialized agents actively interferes with and corrects potential tool-abuse risks in the enhanced pipeline. we provide an overview of our framework in figure 1 , which depicts art as an example of ## experiment setup tasks and datasets. we perform our experiments under the open-domain qa setup @xcite using three benchmark datasets: mintaka @xcite , popqa @xcite , and entity questions @xcite . a histogram of the popularity distribution of these datasets can be found in figure 2 . following the findings from mallen et al. (2023) which point out that retrieval is mandatory when the model lacks parametric memory, we sample the tail distribution of the three datasets in figure 2 to simulate the setting when tool-use agents are dealing with out-of-scope knowledge. specifically, we set dedicated threshold based on each dataset to construct the low popularity subset.appendix a.1 offers a detailed description as well as the augmentation of popularity information of the three datasets. models. experiments are run on two chatgpt models and llama-3-70b-instruct. we select the more advanced gpt-4-turbo as the teacher model and gpt-3. we conduct our study on two tool-use frameworks, dsp @xcite and art @xcite . in addition to the original setting, we also introduce verbalized confidence elicitation settings of the two frameworks denoted as art (v) and dsp (v). in we can see that when smartcal is augmented on both frameworks, it either surpasses or performs on par in terms of qa performance compared to the baseline setting as well as the verbalized calibration setting. the baseline settings of dsp achieves an average of 41.9% on all datasets, while art has an average accuracy of 45.4%. in comparison, smartcal achieves 51.5% when adapted to dsp and 53.0% when adapted to art, with an average advantage of 8.6% in accuracy improvement. we also observe an excessive inferiority in qa accuracy for gpt-3.5-turbo on popqa dataset, where the model is unwilling to answer most questions. we elaborate this observation in appendix a.3.1. ## conclusion in this paper, we identify tool-abuse in llm reasoning, which involves a combination of tool-misuse and degraded tool calibration performance. we also observe a consistently high calibration error regardless of increasing model scales. we then propose a novel framework smartcal to mitigate this issue. to our knowledge, this is among the first efforts to study the topic of recalibration for llm-based tool-use.