Tools Fail: Detecting Silent Errors in Faulty Tools (2024)

Abstract: Tools have become a mainstay of LLMs, allowing them to retrieve knowledge not in their weights, to perform tasks on the web, and even to control robots. However, most ontologies and surveys of tool-use have assumed the core challenge for LLMs is choosing the tool. Instead, we introduce a framework for tools more broadly which guides us to explore a model’s ability to detect “silent” tool errors, and reflect on how to plan. This more directly aligns with the increasingly popular use of models as tools. We provide an initial approach to failure recovery with promising results both on a controlled calculator setting and embodied agent planning.

## introduction tools offer a convenient way to augment capabilities beyond text-based reasoning, from executing code to incorporating recent data through web search, and even facilitating multimodal interactions. while the term "tool" is often interpreted to mean offloading specific deterministic functions to external apis, as tasks grow more complex, the definition is expanding to include learned modules such as translators and object detectors, as well as heuristics-based policies like search algorithms and robotic skills. llms themselves are also being used as tools, particularly as task planners in robotics, chained with vision models and robot policies to perform navigation and manipulation @xcite @xcite @xcite @xcite . as tools take on more responsibilities, assessing and ensuring their reliability becomes crucial; a failure in one tool can trigger a cascade of errors, leading to complete task failure. recent studies have suggested recovery mechanisms, such as correcting inputs based on api error messages @xcite @xcite . however, most methods rely on two underlying assumptions: that accurate inputs guarantee flawless outputs, and that errors are accompanied by explicit signals. yet, real-world scenarios challenge the premises, as failures often arise from unpredictable environmental dynamics and inherent inaccuracies of tools themselves. this paper introduces a taxonomy to categorize sources of tool-related errors and recovery methods. we shed light on the often overlooked case: "toolbased" failures. as opposed to input-based errors which are often accompanied by error messages, most tool failures are "silent." this poses unique reasoning challenges for the llm, which must actively 1. detect the failure, 2. infer the source, and 3. plan recovery strategies. in this paper, we focus on the first step, detection, as it is the prerequisite for downstream fault assignment and recovery. we investigate tool errors in two distinct set-tings (fig. 1 ) -a controlled environment where an llm solves arithmetic problems using a broken calculator, and a more natural "broken" tool setting involving a multimodal instruction-following agent. we investigate whether llms can detect incorrect tool outputs without explicit error signals, to observe overtrusting of tools. motivated by how humans detect tool failures based on internal expectations of correct outputs, we devise three in-context interventions, and find that llms can learn to doubt tools and detect mistakes. following the taxonomy, we further examine how much and what type of deviation is necessary to trigger the llm's recognition of the tool error in each setting. tools text-based tools help compensate for llms' relative weakness in world knowledge and computational precision @xcite @xcite @xcite . multimodal tools allow llms to receive inputs from other modalities and generate grounded answers @xcite @xcite . outputs of vision-language models @xcite , object detectors, ocr models, and speechto-text apis @xcite have been added to the llm's prompt, enabling zero-shot inference on multimodal tasks. ## background notation we outline a typical tool-use scenario in fig. 1a with the following notation: the llm first selects tools and constructs toolspecific arguments i from the task input x. based on the tool result o, the final task prediction ŷ is made. notably, the flexibility of llms as an interface allows tool inputs to be constructed based on enriched context information c throughout the task. c may include task specifics, api docstrings, any external feedback like error messages, or even previous action trajectories in interactive tasks. additionally, we denote the oracle values of the input, output, context as i * , o * , and c * . the tool input i and output o may contain inaccuracies since they are essentially outputs of preceding llm/tool calls. fig. 1b demonstrates a scenario where i contains a mistake (15 x 58 should be 15 * 58). the context c can also be incomprehensive or noisy, as they are approximations of the real world. moreover, the tool t θ can be suboptimal in multiple dimensions. for deterministic apis, a suboptimal tool may have been chosen by an llm @xcite . for learned tools, the tool itself is an inherently imperfect parameterized model, thus t θ . ## error sources the tool output o is accurate if and only if: formally, to obtain o with deviation smaller than ϵ, d(o, o * ), is a union of component error bounds: if any condition above is not met, output errors will lead to task failure. the following sections discuss each condition, and a tools themselves can make mistakes, even when the input or context is perfect. this situation is especially prominent as learnable tools are becoming more widely adopted in practice. llms are prone to generating factually incorrect statements even when reference documents are provided through context @xcite . search apis might fail not because of the input query's clarity, but due to an imperfect database/dense retrieval method. the tool's precision can also contribute to failureheuristic-based search/manipulation robot policies can fall apart when they lack the precision needed to address the complexity of real-world scenarios. due to the absence of explicit error signals, toolbased errors require the tool-using model to reason over indirect cues. in easier cases, errors can be recognized based on well-calibrated confidence scores. much harder cases, however, arise when a tool confidently produces errors. in such scenarios, a broader context may help identify these hidden errors. multiple tools presenting conflicting evidence (e.g., fact verification tool vs search api), disagreement between different modalities @xcite , or prediction inconsistencies over multiple trials @xcite or timesteps @xcite , may help surface potential limitations of the tool. place, and advocate for meta-cognitive reasoning. ## a broken calculator humans use tools with a rough expectation of what correct results should look like, allowing them to spot outputs that are obviously wrong. for example, for multiplying 120 by 131, we can expect a result around 10,000 and ending in zero, even if we don't know the exact answer. if the tool makes arithmetic mistakes, can llms also detect faulty outputs? what is the answer to : (2 + 3) * 5? refer to the tool output below. # calculator api @xmath0: thought : your reasoning process answer : ... # answer ## detecting tool-based mistakes the results in §6 suggest that it is challenging for llms to simultaneously detect and override faulty outputs, even for capabilities that are decently performed without tools. thus, next we narrow the llm's responsibility to "detecting" mistakes. ## natural tool errors: alfred we now consider a setting where tool-based errors occur more naturally via alfred @xcite , an embodied instruction following benchmark. involving language understanding, perception, spatial reasoning, and action planning capabilities, a common approach is to incorporate multiple specialized modules @xcite , as opposed to end-to-end training. multiple modules, or tools collaborating with each other in alfred offer a unique opportunity to study the robustness of llms to various tool errors. as in fig. 1d , the object detector's mistakes are silently passed on to subsequent tools, leading to error cascades in the action planner. in such scenarios, llms that can detect tool errors help improve the system's robustness, by correcting some obvious semantic anomalies @xcite or delegating operations to other tools or humans. in this section, we investigate whether llms can detect these realistic, multimodal tool errors arising from individual modules used in the film architecture @xcite . specifically, we test the llm's fault detection capability on two distinct tools -the object detector and the action planner. ## conclusion we characterize the trust dynamics of modern llms with respect to tool usage. by establishing an extensive taxonomy of tool-related errors and recovery strategies, we identify fundamental challenges associated with integrating learned tools. our experiments span both synthetic and natural tool failures, and affirms current llms' ability to identify silent tool failures. this work paves the way for future research on harnessing llms as sophisticated tool-reasoners.