W hen2 C all: When (not) to Call Tools (2025)

Abstract: Leveraging external tools is a key feature for modern Language Models (LMs) to expand their capabilities and integrate them into existing systems. However, existing benchmarks primarily focus on the accuracy of tool calling—whether the correct tool is called with the correct parameters—and less on evaluating when LMs should (not) call tools. We develop a new benchmark, When2Call, which evaluates tool-calling decision-making: when to generate a tool call, when to ask follow-up questions and when to admit the question can’t be answered with the tools provided. We find that state-of-the-art tool-calling LMs show significant room for improvement on When2Call, indicating the importance of this benchmark. We also develop a training set for When2Call and leverage the multiple-choice nature of the benchmark to develop a preference optimization training regime, which shows considerably more improvement than traditional fine-tuning. We release the benchmark and training data as well as evaluation scripts.

## introduction tool-calling is an increasingly important capability for modern lms as it allows them to connect with existing apis or tools to use real-time information, retrieve information from databases, or carry out actions by integrating with existing systems. this is especially important given recent advances in small language models which can be deployed on devices, as smaller models do not store as much knowledge as larger models, and thus benefit greatly from access to external tools. in a typical setup, the model is provided with a list of tool or api specifications in the system prompt. the model can access these tools by generating one or more tool calls as code (usually json) which conforms to the api specification. any tool calls are intercepted and executed, and the result is returned to the model. in the second step, the model generates a text response to the user based on the tool call result, similar to rag. notably, the tools provided in the system prompt may differ between train and inference time. if a model is not deployed with a tool that answers the question (even if it has seen such tools during training), such as a customer service lm being asked about tomorrow's weather, the lm should say that it cannot answer the question, not hallucinate a previously seen weather tool (or hallucinate tomorrow's weather). more subtly, an lm with access only to a database of student records might be asked to retrieve their grades instead, as shown in figure 1 . a further opportunity for hallucination is that the appropriate tool may be available, but the user may not provide enough information to fill its particular required parameters. in this case, we expect the lm to ask a follow-up question, not hallucinate the missing parameters. feature when2call bfcl toolsandbox toolbehonest older tool(s) provided, one correct ✓ ✓ ✓ ✓ ✓ tool(s) provided but none correct ✓ ✓ ✓ ✓ no tools provided ✓ question missing information ✓ ✓ ✓ tool call validation ✓* ~✓* ✓* quantifies answer hallucinations ✓ quantifies tool hallucinations ✓ ~✓ † quantifies parameter hallucinations ✓ quantifies follow-up questions ✓ we find that modern tool-calling lms of all sizes have much room for improvement on when2call, which is unsurprising given that most publicly available training datasets contain many examples of calling tools when tools are provided in the system prompt but a considerably lesser amount of examples of not calling tools when tools are provided. to address this, we develop a matching training dataset for when2call. we leverage its multiple-choice format to implement supervised fine-tuning (sft) as well as reward-aware preference optimization (rpo) training @xcite and show that rpo training in particular substantially increases performance on when2call benchmark and on bfcl irrelevance while still maintaining competitive scores on the portion of bfcl where a tool should be called. ## multiple-choice evaluation we implement the multiple-choice evaluation using log-probability over the four possible answers to determine the model's choice, rather than having the model generate the choice number (e.g., "answer: (a)"). for tool-calling, the meta-task of selecting among answers may be unnatural for the models, and presentation details such as answer order @xcite @xcite and the number of answers @xcite can artificially affect accuracy. log-probabilities allow us to bypass all of this. we report accuracy, (byte-)lengthnormalized accuracy foot_0 and f1 metrics. we implement our multiple-choice evaluation as a task in lm evaluation harness @xcite , including the preprocessing necessary for evaluating all models in ## llm-as-judge evaluation one limitation of evaluating our multiple-choice benchmark with log-probabilities is that we cannot evaluate closed-source tool-calling models. thus, we implement a second, alternative llm-as-judge metric. by comparing the two metrics, we can also alleviate concerns that the precise answer phrasing chosen by mixtral 8x22b when generating our benchmark might affect model performance. this method prompts the target lm exactly the same as for multiple-choice but has the lm generate a free-form answer instead of getting log-probabilities. we then use an llm-as-judge, specifically gpt-4-turbo-04-09, to classify the target lm's generated output into the four multiplechoice categories: direct answer, tool call, followup question and unable to answer. since we only ask the llm-as-judge to classify the output among these categories, we are not dependent on the judge's own tool-calling (or when-to-call) capability. then, we can calculate the category accuracy (now direct, not normed by length) and f1 score for the multiple-choice version. tool hallucination rate over the questions where no tool is provided can be calculated in the same way as before. we use 100 representative examples from each category of when2call to keep costs accessible, as shown in ## related work with tool-calling growing in popularity (see @xcite for a survey), a number of tool-calling benchmarks have been developed. while some benchmarks break the tool-calling process into subtasks @xcite @xcite , most recent benchmarks treat tool-calling as a single step, a trend which our benchmark follows. in this case, the model must return either a tool call or an appropriate text response. however, most benchmarks only test the case where the correct api is provided and focus either on validating the tool call @xcite @xcite or evaluating the final answer or system state, which assumes a correct tool call @xcite @xcite @xcite . among benchmarks, only bfcl @xcite and toolsandbox @xcite attempt to address cases where the correct api is not provided, or information is missing with their irrelevance and insufficient information categories respectively, which provide no correct tool. toolsandbox and bfcl's multi-turn category also contain some examples of follow-up questions. both of these categories, however, only evaluate whether a tool call is made or not, and not what the model does instead. toolbehonest @xcite also considers these cases but only evaluates the subtask of whether the task is solvable. for training data, the glaive v2 training dataset @xcite is the only publicly available training dataset to include cases where the correct api is not provided, or information is missing, but it does not separate such items from the rest of the training set, or provide an evaluation metric other than exact match / crossentropy loss. other popular training datasets like hermes (interstellarninja and teknium, 2024) and apigen @xcite do not cover these cases. tool hallucinations have only been tackled very recently. @xcite report the tool hallucination rate of their model, granite. the toolbehonest benchmark @xcite studies tool hallucination with an indirect measure by asking models to classify the solvability of tasks ("solvable" indicates that the model is hallucinating that the tool can be used). while the hallucination rate can be calculated manually from the detailed output of bfcl, no previous benchmark provides explicit support for calculating tool hallucination rate, and none study how often the model hallucinates a text answer instead. ## conclusion we presented when2call, a new synthetically generated dataset to evaluate when tool-calling lms should (not) call tools and how they should behave if they can't, using a multiple-choice format over four different types of behavior. we present a traditional accuracy/f1 metric using log-probabilities as well as an llm-as-judge alternative which allows the evaluation of generated outputs, particularly of closed-source models. unlike the main previous benchmark evaluating when not to call tools, bfcl irrelevance, which is already beginning to saturate foot_2 , we find that even large tool-calling models still struggle on our more difficult task, which requires not just not calling a tool, but choosing the correct non-tool-call response, and has a higher similarity between the question and the nonetheless incorrect tool calls. further, we show that achieving the right degree of conservativeness for tool-calling models is not trivial, as simple instruction-tuning on the when2call training dataset in conjunction with other instruction-following and helpfulness data can lead to over-conservative behavior. we propose an rpo training method that leverages the multiplechoice nature of the dataset and strikes a balance between the pressures of when vs. when not to call tools. training on when2call may also lead the model to overly prefer specific answer phrasings, however, impacting its score using log-probability multiple choice method. we recommend using the llm-as-judge method, where possible, after training on when2call to mitigate this. finally, we offer scripts to calculate the confusion matrix and hallucination rate as part of model evaluation, allowing model developers to understand the individual failure patterns of their model beyond a single accuracy score and develop a targeted training regime in response.