Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models (2025)

Abstract: Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks. Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step. However, the performance of IR models in tool retrieval tasks remains underexplored and unclear. Most tool-use benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios. In this paper, we propose ToolRet, a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets. We benchmark six types of models on ToolRet. Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on ToolRet. This low retrieval quality degrades the task pass rate of tool-use LLMs. As a further step, we contribute a large-scale training dataset with over 200k instances, which substantially optimizes the tool retrieval ability of IR models.

## introduction large language models (llms) have demonstrated remarkable progress across various natural language processing (nlp) tasks, such as text summarization @xcite . however, they suffer from inherent inabilities to interact with the physical world and access vast, up-to-date knowledge @xcite . to alleviate these drawbacks, tool learning is proposed to equip llms with external tools, augmenting them as agents to manipulate tools for practical task-solving @xcite . figure 1 : correlation between the tool retrieval performance (e.g., recall@10) of ir models and the end-toend task pass rate of tool-use agents. the initial step @xcite @xcite . this step becomes particularly critical in real-world scenarios where the candidate tools are usually large-scale and many of them are similar in functionality @xcite . however, most existing work @xcite simplifies this retrieval process by manually pre-selecting a small set of 10-20 relevant tools for each evaluation task. for example, the toolace @xcite and toolbench @xcite annotate about 10 tools per task. while recent information retrieval (ir) techniques such as semantic matching @xcite , can assist with tool retrieval, they are often trained on ad-hoc tool-use datasets, lacking comprehensive evaluation on diverse scenarios, especially for unseen tasks. to further explore the importance of tool retrieval, we conduct a pilot experiment on toolbench @xcite . as shown in figure 1 , we observe that (i) the agent's performance substantially drops when replacing the officially annotated toolset with the retrieved tools; and (ii) even strong retrievers like colbertv2 @xcite , struggle to retrieve target tools effectively. these findings highlight the necessity to (i) systematically evaluate ir models on diverse tool retrieval tasks; and (ii) analyze the impact of retrieval on the end-to-end task pass rate. in this work, we introduce toolret, the first large-scale tool retrieval benchmark comprising 7.6k diverse retrieval tasks and a corpus of 43k tools, which comprehensively evaluates ir models across diverse retrieval scenarios. specifically, we collect query-tool datasets from the following sources: (i) tool-use agent benchmarks from published research papers in ai conferences, such as acl and neurips; (ii) related conference resources such as appbench in emnlp and tool-lens in cikm; and (iii) other publicly available datasets from the open-source community, e.g., huggingface. the collected data is carefully curated to cover a wide range of practical tool requirements, comprising diverse types of tool documentation, domains, and varying query lengths. then, we standardize the format of all the collected tasks, aligning them with retrieval tasks similar to the format in mteb, where each retrieval task contains a query and target tools (e.g., labels). to support the instructional retrieval @xcite setting of our benchmark, we also introduce a target-aware strategy to supplement each query with an instruction using the powerful llms (i.e., gpt-4o). we systematically evaluate five types of ir models such as embedding models and llm re-ranking, under various experimental settings. our results reveal that even the best model (i.e., nv-embedd-v1) that demonstrates strong performance in conventional ir benchmarks, achieves an ndcg@10 of only 33.83 in our benchmark. this highlights the challenges of the tool retrieval tasks. we identify two key factors contributing to this performance gap: (i) lower term overlap between queries and target tools in tool retrieval tasks, which demands higher representation abilities for ir models to accurately match query intent with the correct tools; and (ii) task shift from conventional informationseeking tasks (e.g., document retrieval) to tool retrieval, leading to suboptimal performance of ir models that are not explicitly optimized. to enhance the retrieval performance and enable ir models to augment tool-use agents, we further propose the toolret-train, a large-scale training dataset containing more than 200k retrieval tasks. we extend our data collection process from toolret to include the training set of three mainstream tool-use datasets, including toolace @xcite , apigen @xcite and tool-bench @xcite . to enable the training, we pair each retrieval task with 10 negative tools retrieved by the nv-embed-v1. finally, each training example contains the query, an generated instruc-tion, the target tools, and the negative tools. results show that the ir models trained over toolrettrain, exhibit significant improvements in the retrieval process, leading to a higher end-to-end task pass rate when integrated with tool-use llms. our contributions are summarized as follows: (i) we introduce toolret, the first evaluation benchmark for tool retrieval tasks. (ii) we evaluate the tool retrieval performance of various ir models and analyze the impact of retrieval on the end-toend task pass rate of tool-use llms; and (iii) we contribute to a large-scale training dataset that enhances the performance of ir models, improving their ability to augment tool-use llms effectively. ## benchmark construction 3.1 data collection to build a comprehensive benchmark for tool retrieval evaluation, we collect data from the following well-known sources: (i) tool-use llm benchmarks: a wide range of benchmarks published in leading ai conferences such as acl and neurips; given the rapid development of benchmarks from these sources, we collect datasets released between the august 2023 to december 2024 in this version. 3 we download these data from official channels based on their usage requirements and totally collect more than 30 datasets. since the data sources are diverse and their original formats vary substantially, we perform necessary data cleaning operations such as deduplication and text normalization to ensure consistency and quality. we observe that most of the collected datasets are originally designed to evaluate the tool-use capability of llms, where the llm is required to correctly call a sequence of target tools given an input query. to facilitate retrieval evaluation in tool-ret, we align the format of all collected tasks with the well-established ir benchmark like beir and mteb. specifically, each task consists of a query as 3 our team will maintain and update the benchmark. ## benchmark statistic ## retrieval affect downstream task in this section, we qualitatively analyze the impact of retrieval performance on downstream tool-use agents. we conduct end-to-end evaluations on tool-bench @xcite dataset using the official pass rate metric that evaluates whether the model successfully calls target tools to complete a task. ## discussion in this section, we discuss the following three open questions related to the design and reliably of our benchmark construction, as well as potential extension. fine-grained functional differences of tools in this work, we construct the overall tool corpus by merging tools from various existing tool-use benchmarks. this heterogeneous construction strategy inevitably leads to overlapping functionality among different tools, which may raise concerns about unreliable evaluations, such as the one-tomany problem 5 . however, the following reasons support the reliability of our approach: (i) in realworld scenarios, many different but functionally similar tools exist, but typically, only the most suitable one is chosen. in this work, we focus on such real-world scenarios, where the ground truth from the original dataset is considered the most appropriate. we argue that models should have the fine-grained discrimination ability to identify the most appropriate tool; (ii) even if tools appear to have overlapping high-level functionality (e.g., bing search vs. google search), they often differ in important dimensions such as input parameters (e.g., support for language-specific filtering) and specific application scope (e.g., medical search vs. general news search). for example, for a query like search news articles in chinese, a tool like bing_search_with_lang_param is a more precise match than a generic search api without language constraints. in line with prior work @xcite @xcite , our benchmark encourages models to retrieve semantically and functionally appropriate tools, not merely those with similar surface forms. reason for our format-based categorization unlike existing tool retrieval benchmarks @xcite , we introduce a novel task categorization by dividing the overall toolret dataset into three subsets: web apis, code functions, and customized applications. this division is motivated by two reasons: (i) it naturally reflects the structural features of tools collected from over 30 diverse datasets. this categorization aligns with conventions established in prior research on toolaugmented llms, where these three formats are commonly used as the primary categories for executable tools. (ii) this format-based taxonomy provides a clear, interpretable, and effective framework for analyzing model behavior across different tool representations. beyond format-based category in addition to the format-based categorization presented in this work, we believe that other dimensions of categorization can also offer valuable insights into retrieval behavior. therefore, we propose extending the category structure in future work to include three additional dimensions for more comprehensive and customized evaluations: (i) query length (input complexity): tasks are grouped based on used as the label, potentially leading to unreliable evaluations. the query token length, with categories such as 0-25, 25-50, 50-75, and 75+ tokens. (ii) number of target tools (multi-label complexity): queries are grouped by the number of relevant ground-truth tools, categorized as 1, 2-3, @xmath0. (iii) lexical overlap (rouge-l) between query and tool documentation: tasks are grouped by the degree of lexical overlap, categorized as low (<0.05), medium (0.05-0.2), and high (≥ 0.2). a lower overlap indicates a higher need for deep semantic matching between the query and the correct tools. ## conclusion in this work, we introduce toolret, the first diverse tool retrieval benchmark comprising 7.6k queries, each paired with an instruction, and a corpus of 43k tools. toolret is a heterogeneous benchmark, constructed by aggregating existing tool-use datasets and aligning them into a unified format, similar to conventional ir benchmarks such as mteb. we evaluate state-of-the-art ir models on toolret and uncover a surprising finding: even models with strong performance on conventional ir benchmarks struggle in tool retrieval. this low retrieval quality significantly degrades the end-to-end task pass rate of tool-use llms. inspired by this, we further propose tool-ret-train, a large-scale training set containing over 200k retrieval tasks. results show that ir models trained on toolret-train exhibit substantial improvement and also enhance the pass rate of tooluse llms by 10%-20%. in the future, we plan to extend the toolret into multimodal scenarios.