Toolken+: Improving LLM Tool Usage with Reranking and a Reject Option (2024)

Abstract: The recently proposed ToolkenGPT tool learning paradigm demonstrates promising performance but suffers from two major issues: first, it cannot benefit from tool documentation, and second, it often makes mistakes in whether to use a tool at all. We introduce Toolken+ that mitigates the first problem by reranking top-k tools selected by ToolkenGPT and the second problem with a special REJECT option such that the model will generate a vocabulary token if REJECT is ranked first. We demonstrate the effectiveness of Toolken+ on multistep numerical reasoning and tool selection tasks.

## introduction recently, large language models (llm) have been extended by allowing access to external tools such as symbolic computation engines @xcite , databases that serve as external memory @xcite , and others @xcite . tool learning paradigms can be broadly divided into (1) supervised finetuning to leverage tools @xcite , which works well but lacks flexibility and cannot generalize to unseen tools, and (2) in-context learning @xcite , where demonstrations are provided in the prompt; this method is very easy to extend and generalize but often bumps against inherent context length limitations of llms @xcite . toolkengpt @xcite aims to have the best of both worlds: each tool is represented by a special token called toolken that has a trainable embedding and extends the vocabulary. once a toolken has been predicted, the model switches into "tool mode" where it uses in-context examples to fill in the tool's arguments, calls it, sends the results back to text, and returns to language modeling mode. toolkens require very few parameters (toolken embeddings) to be trained while keeping * work was done while at huawei noah's ark lab llm weights frozen, and there is no limit on the amount of data to train these parameters. in this work, we extend toolkengpt with two novel features that aim to fix two important issues. first, toolkengpt cannot use tool documentation known to be helpful for llms @xcite ; we will show that toolkengpt is often unsure which tool to use, and documentation could help decide this. to this end, we introduce a copy of toolken embeddings that rerank retrieved tools, i.e., take top-k tool candidates, prepend the prompt with their documentations, and ask the llm to choose the most relevant tool. second, toolkengpt often makes mistakes in judging when to use tools, calling them too often. to alleviate this, we introduce an extra rej ("reject") "tool" that switches back to text generation without invoking any tools. rej is provided as an option for the reranking mechanism introduced above. the entire operation of toolken+ is illustrated in figure 1 . overall, we aim to minimize false positive errors for tool invocations and tool misclassification rate for toolkengpt. this significantly improves the model's robustness, allowing for developing more trusted llm agents that can have access to a wider variety of tools. a general tool usage paradigm for llms @xcite has four stages: whether to use a tool, which tool to use, infilling the arguments, and dealing with the tool's output. our methods improve the first and second stages of this process. moreover, we provide a formal justification for the toolken training algorithm based on variational inference. we evaluate our results on the gsm8k @xcite , metatool @xcite , and virtualhome @xcite datasets, showing significant improvements. thus, our contributions are as follows: (1) solutions to issues associated with the first two stages of the tool usage process for llms; (2) a theoretically grounded training procedure for the introduced toolken embeddings; (3) an empirical evaluation study supporting the efficiency of our approach. below, section 2 compares our approach with recent related work, section 3 introduces our modifications and formal justification for training and inference, section 4 presents experimental results, and section 5 concludes the paper. natural language feedback. huang et al. (2023a) suggested the idea to use llm feedback as training inputs, using the generated rationaleaugmented answers to fine-tune the model. to alleviate incorrect reasoning steps with tool usage, @xcite suggested to generate natural language feedback from a critic model learned separately. note that although this work also addresses the issue of using incorrect operations in the math world problem task, it requires to train a critic model. overall, our approach adheres to the paradigm of prompted refiners as shown in @xcite @xcite , where the same frozen model is used for reasoning and providing feedback. our method does not invoke the tools themselves and does not work with their outputs, only produces special tokens for tool invocation. in a recent work, @xcite also used model mistake correction to improve the quality of solving math problems. however, their suggested approach relied on gpt-4 output and, again, needed to fine-tune the model. chain of thought reasoning. chain of thought prompting was originally introduced by @xcite ; @xcite to enhance the ability of large language models to perform complex reasoning. @xcite showed that a llm is a good zero-shot reasoner with a simple prompt before each answer. to further improve reasoning skills, @xcite introduced selfconsistency decoding that reranks the generated rationales by taking a majority vote over the final numerical answers. similarly, our approach also benefits from reranking that can use additional information such as tool documentation. recently, @xcite proposed a bootstrapping technique that was able to improve the performance on reasoning tasks even without a massive rationale dataset. in their approach, the model is repeatedly fine-tuned on a dataset of selfgenerated rationales; our approach is similar to this one since bootstrapping is used to train the introduced toolkens but we do not need to tune the llm weights. tool-augmented language models. one direction for augmenting llms with external tools is fine-tuning for tool use @xcite @xcite ; these methods achieve excellent performance but suffer from poor adaptability to unseen tools and high computational requirements to fine-tune the llm. another paradigm learns tool use in context, from documentation and/or demonstrations added to the llm input @xcite @xcite ; these approaches do not require fine-tuning and can learn a new tool given a handful of demonstrations but suffer from performance degradation because of limited context length. in this work, we propose an improvement for the toolken paradigm recently proposed by @xcite that aims to take the best of both worlds. it introduces learnable tool embeddings (toolkens) trained on a dataset of tool use examples while keeping the weights of the llm frozen; it also leverages in-context learning to fill in tool arguments. ## method toolkengpt. toolkengpt introduces an embedding for each tool concatenated with the language modeling head. formally, the embedding matrix w @xmath0, where v is the original vocabulary of tokens and d is the latent dimension, is extended with a matrix w @xmath1, . . . , t |t | }, and the next token probability for the augmented llm is calculated as inference is divided into two interleaving stages: reasoning mode, where the model generates a rationale using p aug (x i |@xmath2, and tool mode, where it infills tool arguments given a prompt with usage examples. thus, w t are learned by solving toolken+. first, we extend the tool set as t ′ = t ∪ {rej}, where rej is a special tool responsible for switching back to reasoning mode. second, instead of taking the best proposed tool we take the retrieved subset of top k tools t k ⊆ t ′ and ask the model to choose one. the proposed toolken+ model takes the previously generated sequence and top k tools retrieved by toolkengpt t k as input and is asked to generate a tool from t k ∪ {rej}. formally, toolken+ produces p rank ( approximate inference. the next token probabilities in algorithm 1 are given by ## conclusion in this work, we have proposed an improvement for the toolkengpt approach of learning special token embeddings that adds a reject option and a reranking mechanism for tool selection. our approach significantly improves the results via in-context learning while still keeping llm weights frozen and learning only toolken embeddings. the considered approach is an important step towards improving the robustness of ai agents and user-facing tools based on modern llms. better tool use not only makes the tool usage results more robust but also has a potential to reduce hallucinations and make llm answers more trustworthy by allowing an llm to reliably run external tools to verify its answer; this is a very important consideration in practical usage. in future work, we hope to extend the toolken+ approach to other external tools and/or agents to further expand the capabilities of modern llms.