Making Language Models Better Tool Learners with Execution Feedback (2024)

Abstract: Tools serve as pivotal interfaces that enable humans to understand and reshape the environment. With the advent of foundation models, AI systems can utilize tools to expand their capabilities and interact with the real world. Existing tool learning methodologies, encompassing supervised fine-tuning and prompt engineering approaches, often induce large language models to utilize tools indiscriminately, as complex tasks often exceed their own competencies. However, introducing tools for simple tasks, which the models themselves can readily resolve, can inadvertently propagate errors rather than enhance performance. This leads to the research question: can we teach language models when and how to use tools? To meet this need, we propose Tool leaRning wIth exeCution fEedback (TRICE), a two-stage end-to-end framework that enables the model to continually learn through feedback derived from tool execution, thereby learning when and how to use tools effectively. Experimental results, backed by further analysis, show that TRICE can make the large language model selectively use tools by improving the accuracy of tool usage while enhancing insufficient tool learning and mitigating excessive reliance on tools.

## introduction the recent rapid advancement of foundation models @xcite @xcite @xcite makes it practical for ai machines to create @xcite and utilize tools effectively @xcite , which greatly transcends their inherent limitations in various underlying areas, including arithmetic @xcite , knowledge updating @xcite , multi-modal semantic analysis @xcite ## related work tool learning. though possessing remarkable capabilities @xcite , llms still struggle in many basic aspects where much smaller and simpler tools may precisely excel. under this circumstance, a new paradigm, called tool learning @xcite , is born to combine the strengths of both llms and specialized tools. some works @xcite @xcite regard llms as a decision-making hub for compositional tool using which can be called tool-oriented learning @xcite , while others @xcite @xcite treat tools as complementary resources to extend the power of llms which can be called tool-augmented learning @xcite . despite their success, tool-augmented approaches tend to force llms to use tools mindlessly regardless of whether they actually need tools for help. this may, in some scenarios, steer lms to erroneously choose the type of tools or the way to use tools, making the loss outweighs the gain. compared to previous works, we try to make lms better tool learners by teaching them to use tools selectively instead of blindly. learning from feedback. an intuitive approach of tool learning is to fit llms on examples with human-labeled tools directly @xcite . however, this is often impractical to annotate every possible scenario @xcite and difficult to generalize to new ones. it is worth noting that humans generally have the ability to correct and reflect on their own behavior from instruction: given a math problem, solve it and you can use a calculator for help. input: jerry had 7 action figures and 2 books on a shelf in his room. later he added 4 more books to the shelf. how many more action figures than books were on his shelf? output calculator(7-2-4) llm label (from chatgpt) l clone 1 7 1 reward strategy 1 gold answer execute execution results trial and error @xcite . intuitively, feedbacks from the environments or humans enable llms to understand the impact of their actions and adapt their behavior accordingly. reinforcement learning (rl) excels at enabling models to learn decision-making abilities in complex environments through feedback @xcite @xcite . ouyang et al. (2022) apply a state-of-the-art rl algorithm, ppo (schulman et al., 2017), to align llms with human feedback. liu et al. (2022) reinforce knowledge for commonsense question answering with a fixed qa model providing feedback. yuan et al. (2023); lee et al. (2023) offer a promising alternative that leverages powerful off-the-shelf llms to generate preferences. compared to the previous feedback framework, we introduce rlef for tool learning which reinforces the llms with the execution result of tools. ## conclusion in this paper, we focus on addressing the challenge of selective utilization of tools by llms and propose a two-stage end-to-end training framework dubbed trice to make llms better tool learners with execution feedback. through comprehensive experiments, we show that our method can achieve better performance compared to gpt-3.5. further analyses illustrate that trice can selectively use tools by improving the accuracy of tool usage while enhancing insufficient tool learning and mitigating excessive reliance on tools.