LLM s in the Imaginarium: Tool Learning through Simulated Trial and Error (2024)

Abstract: Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM’s ‘imagination’ to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.

## introduction tools play an essential role in extending humans @xcite and other animals @xcite beyond the confines of physical bodies to better perceive and exert impact on their environment. there is a recent surge of interest in augmenting large language models (llms) with tools to transcend the confines of their static parametric knowledge and text-in-text-out interface, empowering them to acquire up-to-date information, call upon external reasoners, and take consequential actions in external environments @xcite @xcite . existing work on tool-augmented llms primarily aims to increase the ease of adding new tools, or the ability to access many tools (e.g., up to 16,000 apis @xcite ). this is achieved through one of two common approaches: 1) in-context learning (icl), which prompts frozen llms with api specification and tool use examples (i.e., instruction-api call pairs) @xcite @xcite , or 2) fine-tuning with tool use examples synthesized by llms @xcite @xcite . while coverage and flexibility are important for tool use, a critical aspect that, perhaps surprisingly, has been understudied is simply how accurately an llm uses tools for which it has been trained. icl is flexible but hard to drive to production-level accuracy. fine-tuning can potentially lead to better accuracy by integrating a larger number of examples, but existing work mostly focuses on generalizing to unseen tools instead of optimizing an llm's ability to use tools seen during training @xcite @xcite . meanwhile, practical deployment of tool-augmented llms necessitates a high level of accuracy as they enable consequential actions, e.g., financial transactions or other legally binding operations. inaccurate tool use could lead to undesired or harmful outcomes and quickly undermine user trust. how to truly master a tool? we turn to successful precedents in the biological system such as humans @xcite , apes and corvids @xcite . learning to use a tool is a rather advanced cognitive function that depends on many other cognitive functions. first of all, trial and error is essential for tool learning @xcite . we do not master a tool solely by reading the 'user manual'; rather, we explore different ways of using the tool, observe the outcome, and learn from both successes and failures. furthermore, intelligent animals do not just do random trial and error-we proactively imagine or simulate plausible scenarios that are not currently available to perception for exploration @xcite . finally, memory, both short-term and longterm, is instrumental for the progressive learning and recurrent use of tools @xcite @xcite . to this end, we propose simulated trial and error (ste; illustrated in figure 1 ), a biologically inspired method for tool-augmented llms. given a tool (e.g., an api with its specification), ste leverages an llm to simulate, or 'imagine', plausible scenarios (i.e., instructions) for using the tool. it then iteratively interacts with the api to fulfill the scenario by synthesizing, executing, and observing the feedback from api calls, and then reflects on the current trial @xcite . we devise memory mechanisms to improve the quality of the simulated instructions. a short-term memory consisting of recent trial and error trajectories is employed to facilitate deeper exploration in a single episode, while a long-term memory containing distilled past exploration and reflections maintains progressive learning over a long horizon. in the exploitation stage, one can use the tool use examples from the explored trials to fine-tune an llm, or simply do icl by retrieving from those examples. we conduct comprehensive experiments on apis from toolbench @xcite and summarize the main findings as follows: fine-tuning (with or without ste) brings the challenge of catastrophic forgetting, where learning new tools may cause the llm to lose its existing tool use capabilities or general language capabilities. we demonstrate that a simple experience replay strategy @xcite could largely mitigate the issue, allowing the api name: forecast_weather description: forecast weather in the upcoming days. your input should be … ## simulated trial and error we introduce our proposed simulated trial and error (ste) for tool learning. ste consists of an exploration phase and an exploitation phase, which are discussed next. ## conclusions motivated by how humans master tools through continual interaction and reinforcement, we propose simulated trial and error, an llm toollearning method built upon progressive memorybased trial and error. experiments on apis drawn from toolbench show the effectiveness of the proposed method, and also that rehearsal-based finetuning could enable continual learning of new tools with preserved previous skills.