Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval (2024)

Abstract: Recent advances in large language models (LLMs) have enabled autonomous agents with complex reasoning and task-fulfillment capabilities using a wide range of tools. However, effectively identifying the most relevant tools for a given task becomes a key bottleneck as the toolset size grows, hindering reliable tool utilization. To address this, we introduce Re-Invoke, an unsupervised tool retrieval method designed to scale effectively to large toolsets without training. Specifically, we first generate a diverse set of synthetic queries that comprehensively cover different aspects of the query space associated with each tool document during the tool indexing phase. Second, we leverage LLM’s query understanding capabilities to extract key tool-related context and underlying intents from user queries during the inference phase. Finally, we employ a novel multi-view similarity ranking strategy based on intents to pinpoint the most relevant tools for each query. Our evaluation demonstrates that Re-Invoke significantly outperforms state-of-the-art alternatives in both single-tool and multi-tool scenarios, all within a fully unsupervised setting. Notably, on the ToolE datasets, we achieve a 20% relative improvement in nDCG@5 for single-tool retrieval and a 39% improvement for multi-tool retrieval.

## introduction recently, large language models (llms) have demonstrated impressive capabilities on a variety of complex tasks, including math, reasoning and coding @xcite @xcite . they can even surpass average human performance on standardized exams such as college entrance tests, law school admission, and math competitions @xcite . however, llms are pre-trained on a static corpus, limiting their * correspondence to: {yanfeichen, jinsungyoon, chenyulee}@google.com i'm planning a trip to france in the summer and i want to dedicate my time there to improve my french language skills, particularly focusing on and cultural understanding. tool name: travel_assistant description: provide personal travel recommendations on itertinary, flights and accommodations. ## related work tool retrievals for tool-use. react (yao et al., 2023) pioneers the interaction and reasoning with diverse tools using in-context reasoning traces, particularly in decision-making and multi-step reasoning environments. @xcite proposes a self-supervised training method with api demonstrations. @xcite and @xcite demonstrate that augmenting llms with tool document significantly improves their ability to generate correct api calls by mitigating hallucinations, compared to prompting with demonstrations alone. @xcite also shows unifying tool instruction leads to better tool usage. however, tool document retrieval for llm tool learning is currently under-explored, as most work simply uses llm agents to retrieve a limited number of tools. @xcite first demonstrate that llms generate more reliable outputs with the integration of a retrieval system using bm25 @xcite and gpt-index @xcite , but still introduce more hallucination and errors compared to the ground truth retriever. some works @xcite @xcite train a sentence-bert transformer model using the fully labeled query-api document pairs as a tool retriever. the key distinction between our approach and existing tool retrieval systems lies in our emphasis on zero-shot usage, eliminating the need for any labeled data. ## method: re-invoke we formulate the tool retrieval task as retrieving the most relevant tools that a downstream agent can execute to fulfill user queries, given a list of tool documents describing the intended tool usage. re-invoke, our proposed fully unsupervised retrieval method designed for tool retrieval tasks, is illustrated in fig. 2 . it consists of two core components: (1) query generator: for each tool document, llms generate diverse synthetic queries answerable by the corresponding tool. these queries enrich the tool document and are then indexed by encoding them into the embedding space when the tool documents are ingested offline. (2) query intent extractor: during online inference, llms extract the core tool-related intent(s) from user queries, filtering out irrelevant background context. each user intent is then encoded into the same embedding space as the tool documents for similarity matching. pseudo-code of re-invoke is described in algorithm 1. ## discussions 6.1 ablation studies re-invoke component evaluation. in this study, we evaluate the tool retrieval performance on each individual re-invoke components: query generator and intent extractor using vertex ai text embedding api. the results in ## conclusion in this work, we present re-invoke, a fully unsupervised tool retrieval approach designed to scale llm tool learning to large toolsets. we leverage llms to enhance tool document context with diverse synthetic queries, extract essential tool-related intents into executable requests through intent extraction. re-invoke offers a fresh perspective on scalable tool retrieval, prioritizing context enhancement and intent understanding without any training data. intent extractor. re-invoke relies on in-context learning and llm's internal knowledge to extract tool related intents. future work could include using the downstream agent's execution results as a feedback to refine intent extraction.