T ool R e AG t: Tool Retrieval for LLM -based Complex Task Solution via Retrieval Augmented Generation (2025)

Abstract: Artificial intelligence agents when deployed to solve complex problems, need to first decompose the task into smaller manageable sub-tasks, and further associate tools if one is required to solve the sub-task. If the size of the set of tools to chose from is large, a retrieval system is usually employed to narrow down the tool choices before the LLM can proceed with associating tools to the sub-tasks. This paper focuses on the retrieval problem to identify the set of relevant tools to solve a complex task given a large pool of tools to chose from using retrieval augmented generation (RAG) and we refer to it as ToolReAGT. The proposed approach employs ReAct prompting to perform the retrieval in an iterative fashion to first identify if a tool is required and then associate one or more tools for each sub-task. This deviates from conventional RAG where an n-best list of tools are identified given the complex task directly. Experiments are presented on the UltraTool benchmark corpus with 1000 complex tasks and over 2000 tools to select from. A conventional RAG-system is established as baseline and compared to the ToolReAGt approach, resulting in an 8.9% improved retrieval accuracy score recall@5.

## introduction the ability of current ai systems utilizing the language understanding and reasoning capabilities of llms in combination with individual tools designed for solving specific tasks, has greatly expanded the application range and capacity of these systems @xcite @xcite @xcite @xcite @xcite . tools which can provide tailored information such as up-to-date temperature measurements, compute mathematical equations or identify objects in images can each contribute to solve an overarching complex task and result in powerful ai agents. however, the efficacy of these systems in completing complex tasks heavily relies on their ability to accurately select appropriate tools for solving individual sub-tasks. figure 1 illustrates the flow diagram of a system that can solve a complex task. the planner is internally made of a task decomposition module that decomposes the complex task into smaller manageable sub-tasks, the tool selection module can dive into a database of tools and retrieve the relevant tools that can solve the sub-tasks and a tool calling module that can compose and call the retrieved tools in a specific order to solve the complex task, which we refer to as a solution. every module can introduce dependencies that the other modules need to adhere to in deriving a solution to solve the complex task. in this paper, we focus on the tool selection module and investigate tool retrieval in depth, with the aim to support the planner in solving complex tasks. the main challenges in tool retrieval are: a) understanding the requirements of the task to be solved and formulating an adequate query to find a suitable tool, b) comprehending the functionality of a tool from its description, and c) ability to distinguish between similar tools to choose the most suitable one. in this paper, we introduce a training-free retrieval-augmented generation @xcite ) architecture called toolreagt. the proposed approach employs react prompting to enhance retrieval using iterative refinement of the prompts. we will also present investigations on the importance of context information when solving a complex task. we present our investigations using the ultratool @xcite benchmark corpus that has over 2000 tools to choose from and compare with traditional rag approaches as well as more recent iterative based approaches that also involve training the retriever. we show through experiments on the ultratool corpus that the proposed training free approach can outperform existing methods. the rest of the paper is organised as follows: an overview of related work is presented in the next section, followed by the description of the conventional rag system and the toolreagt model. then, the benchmark corpus is described and the set-up of the evaluation which is followed by the presentation of results and their discussion, and finally conclusions. ## related work a simple and straight forward approach to selecting the relevant tools is to provide all the tool description in the prompt @xcite @xcite , which can be further combined with fine-tuned retrieval systems @xcite @xcite . but a major limitation of these methods is when the pool size of the tools to chose from increases drastically, that limits to include all the tool descriptions into the prompts. a potential solution is to first build a smaller pool of relevant tools using rag and then proceed to solving the complex task, which is one of the motivations for the method presented in this paper. in @xcite , the authors propose to leverage reinforcement learning to enhance the alignment between user queries and tools in llms. this method focuses on retrieving n-best tools related to the query using query re-writing. it also requires training the retriever using reinforcement learning. in contrast the proposed method follows an iterative prompt refinement and is more focused on solving a complex task, where tools are retrieved in a step-by-step fashion. also, we follow a training free approach. an adaptive truncation of retrieval results is presented in @xcite which treats seen and unseen tools differently to ensure more relevant tools are prioritized. additionally, it introduces a hierarchy-aware reranking which refines retrieval results by concentrating them for single-tool queries and diversifying them for multi-tool queries. while the adaptive truncation method effectively manages unseen tools, our method explores the use of the react framework that inherently performs re-ranking of the relevant tools, but also explores the use of a varied context during retrieval at (a more fine-grained) sub-task level. an approach in which fine-tuned llms are used to capture relationships between user queries and tool descriptions is introduced in @xcite . the method constructs bipartite graphs among queries, scenes, and tools, and it uses a dual-view graph collaborative learning framework to capture intricate collaborative relationships among tools. in this work, we assume that the planner is looking into the relations between tools, where the tool retriever is one of the components of the planner. this is done in a training free fashion and it should generalise to unseen tasks. in @xcite , authors propose iterative llm feedback to improve tool selection, but use a trained dense retriever without the rag-specific generation part. in our method we avoid training the retriever and the iterative refinement is done through a react-agent. ## methods we will first present the general rag architecture and how it can be employed to perform tool retrieval, which we will refer to as conventional rag and is used as a baseline in our study. further we introduce the proposed toolreagt method and present the design changes that are introduced contrasting with the conventional rag. ## data for evaluation, we used the ultratool @xcite corpus which provides a rich number of complex tasks (5824) from 22 domains (e.g. finance, travel, documents, etc.) with a large tool set of 2032 tools. the corpus comes divided into a test set (1000 tasks) and a development set (4824 tasks). for evaluation we employ the 1000 tasks test set which consists of 436 tools (test-436). in addition, we perform evaluation on the same test set using the full (test+dev) 2032 tool set (test-2032). the test tasks include an average number of 2.4 tools per task with the following distribution: 1: 188, 2: 496, 3: 205, 4: 83 and the remainder @xmath0. a major reason to choose ultratool benchmark is that, it has annotations with reference solution plans, i.e. decomposition into sub-tasks, including tool-requiring sub-tasks and their respective tools, enabling objective evaluation via retrieval metrics. this helps us to investigate how the retrieval performance can vary when task decomposition from planner is available apriori. solution plans contain on average 12.1 steps which means there is a high proportion of tool-free steps. for brevity, we chose the english version of the corpus which was originally collected in chinese. contrary to ultratool's original evaluation methods which encompass planning, tool creation and tool usage, we are using the corpus for evaluating tool selection performance of rag-systems with and without using the provided sub-tasks. ## evaluation setup the experiments conducted in this study employ the llamaindex-framework 1 for implementing the rag pipeline. the basic workflow to create a rag-system contains the preparation of source data from which information will be retrieved (typically in the form of documents, i.e. tool descriptions here), ingesting this data into a vector index leveraging an embedding model, defining a query agent together with an llm, and formulating input prompts. these steps will be described next. ## conclusion the paper presented a training free retrieval-augmented generation architecture called tool-reagt to improve tool retrieval performance in the framework of solving complex tasks. the proposed approach employed react prompting to perform an iterative and targeted retrieval of tools, is able to run with and without given task decomposition and showed that the retrieval performance improved on the ultratool benchmark. it is also clearly evident that having access to the task decomposition in advance can greatly benefit the retriever in identifying the relevant tools. results showed the advantage of the proposed approach against conventional rag as well as against other methods in literature that also followed an iterative approach to solving tool retrieval.