GEAR : Augmenting Language Models with Generalizable and Efficient Tool Resolution (2024)

Abstract: Augmenting large language models (LLM) to use external tools enhances their performance across a variety of tasks. However, prior works over-rely on task-specific demonstration of tool use that limits their generalizability and computational cost due to making many calls to large-scale LLMs. We introduce GEAR, a computationally efficient query-tool grounding algorithm that is generalizable to various tasks that require tool use while not relying on task-specific demonstrations. GEAR achieves better efficiency by delegating tool grounding and execution to small language models (SLM) and LLM, respectively; while leveraging semantic and pattern-based evaluation at both question and answer levels for generalizable tool grounding. We evaluate GEAR on 14 datasets across 6 downstream tasks, demonstrating its strong generalizability to novel tasks, tools and different SLMs. Despite offering more efficiency, GEAR achieves higher precision in tool grounding compared to prior strategies using LLM prompting, thus improving downstream accuracy at a reduced computational cost. For example, we demonstrate that GEAR-augmented GPT-J and GPT-3 outperform counterpart tool-augmented baselines because of better tool use.

## introduction recently there has been a surge in research on augmented language model @xcite , which aims to enable models interface existing "tools" for various purposes, such as accessing the latest information @xcite , interacting with third-party services @xcite , performing precise calculations @xcite , or reasoning via code @xcite . the paradigmatic framework of these tool-augmented lm studies generally comprises two steps: selecting a tool and executing it via a generated api call. consequently, choosing suitable tools is essential for task success. ## related work we divide the notable prior works on toolaugmented models into two groups based on how they modify language models: one uses fine-tuning, while the other uses in-context prompting. we also touch upon works in embodied lm applications. ## gear: generalizable and efficient augmented tool resolution we start with the formal problem statement. we are given an input query q that we aim to solve. in addition, we are provided with a tool library t ≜ {(t 1 , d 1 , π 1 ), (t 2 , d 2 , π 2 ), • • • , (t n , d n , π n )} with n tools. each tool t i can receive an api call (e.g., a question or a formula) and respond accordingly, often in the form of natural language. if the provided input is unparsable to the tool, it would return an empty response. each tool is also supplied with its natural language description (d i ) and demonstrations (π i ) showing examples of natural language questions parsed by each tool. gear aims to find the most appropriate tool for solving q. as it can be observed in the algorithm 1, gear iterates over the tools (line 2) and scores each tool i with respect to the given question q (line 5). this score is a linear combination of two scores, a semantic similarity score s(., .) and a pattern similarity score p (., .). semantic score (defined in §3.1) provides a measure of semantic alignment between the tool description d i and the given query q. pattern similarity score (defined in §3.2) scores the alignment between the responses obtained from slm and each tool, which provides an indication of how closely the tool's output aligns with a preliminary answer. the algorithm ultimately picks the most appropriate tool based on their scores (line 7) and obtains the final tool response via an api call generated by a llm (line8, line9). ## experimental findings we compare the downstream performances of models ( §5.1), and compare their generalizability to new tools or tasks ( §5.2). ## conclusion in this paper, we introduce gear: a generalizable query-tool grounding algorithm that enables efficient tool groundings without extra fine-tuning or task-specific demonstrations. this is accomplished by introducing a fine-grained scoring mechanism that leverages both semantic and pattern similarities and leveraging smaller language models for query-tool grounding. to validate the generalizability of gear, we conduct extensive experiments that demonstrate its capacity to deal with large tool libraries and novel tasks.