Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenario (2024)

Abstract: Current research on tool learning primarily focuses on selecting the most effective tool from a wide array of options, often overlooking cost-effectiveness, a crucial factor in human problem-solving. In this paper, we address query routing for homogeneous tools by predicting both their performance and the associated cost required to accomplish a given task. We then assign queries to the optimal tools in a cost-effective manner. Our experimental results demonstrate that our method achieves higher performance at a lower cost compared to strong baseline approaches.

## introduction tool learning @xcite aims to arm large language models (llms) @xcite @xcite with real world tools, to alleviate hallucinations @xcite of llms. existing tool learning methods focus on routing a query to the most effective tool from a large number of options @xcite , while overlooking the crucial aspect of costeffectiveness, which is a significant criterion in human problem-solving. to bridge this gap, query routing for homogeneous tools has become an important issue. taking the retrieval-augmented generation (rag) @xcite scenario as an example. given a user query, a rag system first uses a search tool to retrieve the background documents. then, a llm reads the query and documents to give the response. given the availability of various web search tools, such as google and bing, the choice of which tool to use for a given query becomes an important consideration. first, a query that cannot be solved by one tool might be resolved by another. as shown in figure 1 , our dataset indicates that (1) in 4.5% of cases, directly utilizing the llm to answer queries surpasses the performance of the bing-based rag. and (2) there are 8.1% of instances where the bingbased rag fails to deliver a solution, whereas the google-based rag successfully provides one. furthermore, different tools come with varying usage costs. from a cost-effectiveness perspective, it is preferable to select the least expensive tool that can effectively solve the query. if we can adaptively allocate queries to the optimal tool, we can achieve a balance between cost and performance. additionally, leveraging the complementarity between tools offers the potential for enhanced overall performance. the key challenge in achieving this goal is estimating the performance of candidate tools in handling user queries, instead of making decisions after calling all tools for user queries @xcite . to overcome this challenge, we automatically construct training data including queries submitted to all tools, along with the returned scores. the training data allows us to learn neural models to predict the performance that llm calls each tool to solve each query. in this work, we explore query routing for homogeneous tools and instantiate it in the rag scenario. we propose a router that assigns input queries to the most appropriate search tool without directly accessing any of the tools. to achieve this, we de-velop an automated pipeline to construct training data without manual effort. finally, we design various assignment strategies to allocate user queries to the optimal tool as needed. in summary, we make the following contributions. (1) we propose a general approach that dynamically assigns an input query to the optimal tool from a set of homogeneous candidates, only based on the input query. (2) as far as we know, we are the first to consider the selection of homogeneous tools. our framework is generalizable to any type of homogeneous tools, extending beyond the rag scenario. (3) we evaluate our approach on several qa tasks with multiple llms. experiments show that our method outperforms the use of a single, fixed search tool across different datasets. llms selection earlier, different llms charged differently due to their uneven performance. practitioners focus on effectively choosing llm to save costs. @xcite gradually employs more expensive lms until a satisfactory performance is obtained. @xcite @xcite propose the neural routing function that can precisely distribute each query to the expert llms. however, recently, the cost of using llms has continued to decrease. the usage cost of tools has become a major obstacle to llms-based applications. thus, we explore the selection of homogeneous tools to achieve a balance between performance and cost. ## conclusion in this work, we investigate the problem of selecting homogeneous tools for the sake of costeffective trade-offs. we provide a clear definition of the problem and propose a general framework that can be adapted to any type of homogeneous tools. the experiment shows that our method is effective and can achieve higher performance while at a lower cost compared to strong baselines. this indicates the potential of our framework where multiple homogeneous tools are available.