T ensor O pera Router: A Multi-Model Router for Efficient LLM Inference (2024)

Abstract: With the rapid growth of Large Language Models (LLMs) across various domains, numerous new LLMs have emerged, each possessing domain-specific expertise. This proliferation has highlighted the need for quick, high-quality, and cost-effective LLM query response methods. Yet, no single LLM exists to efficiently balance this trilemma. Some models are powerful but extremely costly, while others are fast and inexpensive but qualitatively inferior. To address this challenge, we present TO-Router, a non-monolithic LLM querying system that seamlessly integrates various LLM experts into a single query interface and dynamically routes incoming queries to the most high-performant expert based on query’s requirements. Through extensive experiments, we demonstrate that when compared to standalone expert models, TO-Router improves query efficiency by up to 40%, and leads to significant cost reductions of up to 30%, while maintaining or enhancing model performance by up to 10%.

## introduction large language models (llms) have demonstrated remarkable performance across a diverse set of challenging domain-specific tasks @xcite . however, no single llm can outperform all others across every task and use case @xcite . recent works @xcite @xcite highlight the urgent need for efficient tools that can unify the expertise of multiple llms, combining them into a single cohesive unit. such tools can allow enterprises to develop applications, e.g., for customer support, on top of a single endpoint that can integrate multiple domain experts and intelligently route any query to the most suitable expert. however, due to the high costs and latency involved in querying llm experts hosted at various providers @xcite , it is essential for such multi-llm querying tools to efficiently and economically direct queries to the most suitable expert. this requires balancing three key factors: query throughput, monetary cost, and model performance -a challenge we refer to as the multi-llm routing trilemma. our aim is to provide an empirical solution to this trilemma by showcasing the potential of a multi-llm routing system that improves this balance. we propose an llm routing system, called tensoropera-router (hereinafter referred to as to-router), to explore the feasibility of building a multi-llm routing model that leverages the collective power of multiple llm experts. to-router aims to efficiently, inexpensively, and accurately answer query prompts by selecting the most costeffective and suitable llm from a diverse set of expert models. our contributions are as follows: ## tensoropera router system overview to effectively learn and deploy a multi-llm routing model, a sequence of different critical development phases need to be executed, from data preparation to router model training and evaluation and model deployment/serving. the proposed to-router system's end-to-end pipeline shown in figure 1 facilitates the development of these phases and in practice has helped to swiftly develop, prototype and deploy different model routing methods into real-world settings. 1 phase 1: router data preparation. the generation of the training and testing dataset for the routing model is a multi-step process. first, we need to find the appropriate domain specific (e.g., bio, coding, physical sciences) instruction datasets and model experts to which we want the routing model to learn propagating relevant query prompts. thereafter, we perform a forward pass over each expert model (step 1) to collect the associated metrics required to train and test the performance of the routing model and create the experts prediction dataset (step 2), i.e., for every prompt in each dataset we query each expert individually. in this work, we collect the following metrics per instruction prompt: {negative log likelihood, bert similarity score (bertsim), inference time in seconds, total input tokens, total output tokens}; for more details on these metrics, please see section 4.4. once the expert prediction dataset is created, we select one of the collected metrics to generate soft labels (step 3) and prepare the final training and testing dataset for the routing model (step 4). in the current work, we use the bertsim scores to create soft labels and train the routing expert model classifier. we use soft labels, since we want the routing model to learn the ranking of the experts in terms of their prediction performance. to generate the soft labels of each expert model and for each instruction record, we pass the selected metric (e.g., similarity score, log loss), through a softmax function with temperature. for instance, for the r-instruction record, the expert (class) softmax probability ϕ r is given by: phase 3: router deployment. when the final routing model is trained, the model is deployed as a standalone endpoint on the platform (step 7), ready to receive user queries (either through cli or web interface). whenever a new user query is submitted, the router first tokenizes and encodes the text of the incoming query prompt using the tuned embedding model from phase 2 (step 8). subsequently, the router performs a forward pass over the trained/finetuned classification model (e.g., mlp, bert) and predicts the most relevant expert model (step 9). depending on which expert model the classification model predicts, the router selects the respective expert-prompt adaptor to submit and execute the query. once query execution completes, the router receives the reply from the expert model and forwards it back to the end user (step 10). throughout the router's deployment time, the platform provides the necessary monitoring capabilities to troubleshoot and tune the routing model, such as number of requests, queries' semantic context, expert models hitting frequency, and total costs. ## experiments in this section we discuss the expert models, benchmark datasets, routing methods and metrics we considered to evaluate the to-router system. ## conclusion we presented for the first time our multi-llm routing system, called to-router. through the to-router system, users can easily interact with multiple llm expert models hosted at the same or across multiple platform providers, without having to restrict themselves to a single monolithic llm system. at the same time, users can overall benefit from significant cost savings (up to 30%) and improved query response times (up to 40%) while maintaining or enriching (up to 10%) model performance. as part of our immediate future plan we aim to evaluate the feasibility of dynamically adding and removing model experts during router's endpoint deployment, and test the routing efficacy of small and large language pre-trained models. finally, we also plan to evaluate approaches where we combine the responses of top-k experts into one instead of returning the response of a single expert.