M ix LLM : Dynamic Routing in Mixed Large Language Models (2025)

Abstract: Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency. Given mixed LLMs with their own strengths and weaknesses, LLM routing aims to identify the most suitable model for each query in the stream to maximize response quality and minimize cost and latency. However, the challenges involve: (1) dynamic trade-offs among quality, cost, and latency; (2) enabling continual learning in deployed systems; and (3) navigating a varying (e.g., new LLM addition or old LLM removal) set of LLM candidates over time. To bridge these gaps, we develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment. Specifically, we first leverage query tags to enhance query embeddings for the routing task. Next, we design lightweight prediction models to estimate the response qualities and costs of queries over LLMs. We then devise a meta-decision maker to choose the query-LLM assignments to best tradeoff response quality, cost, and latency. Finally, the system benefits from continual training, allowing it to adapt to evolving queries and user feedback over time. Our extensive experiments show that MixLLM achieves the best trade-offs in response quality, cost, and latency (97.25% of GPT-4’s quality at 24.18% of the cost under the time constraint).

## introduction large language models (llms) have exhibited abilities to understand massive texts, generate actionable knowledge, enable contextual reasoning, and innovate diverse applications @xcite @xcite @xcite . however, deploying llms presents unique challenges in managing computational resources, optimizing response times, and ensuring scalability. as shown in figure 1 , the diversity of available llms @xcite @xcite @xcite , each with different strengths and weaknesses, poses a challenge when selecting the most appropriate model for a given task. more powerful models, such as gpt-4 @xcite , can deliver high-quality responses, but the pricy cost and computational requirements limit their accessibility. thus, llm routing, which chooses the most suitable llms for incoming queries in mixed llm candidates, is needed to balance the trade-offs between response quality, cost, and latency. existing llm routing methods can be categorized into non-predictive methods and predictive methods. non-predictive methods, like cascading @xcite , firstly exploit smaller llms and then switch to larger llms based on a reviewer model, but this increases both cost and latency as multiple llms are involved @xcite . predictive methods predict the performance of candidate llms to select the best one for each query. for example, hybridllm @xcite exploits a binary classifier to predict the query difficulty for routing, routerbench @xcite predicts response quality directly, and forc @xcite optimizes quality and cost at the set level. however, the challenges involved in existing work are multifaceted. firstly, a key limitation of current methods is the lack of consideration of high latency when too many queries are routed to the same llm. ignoring latency can create bottlenecks, reduce system efficiency, and impact user experience. secondly, continual learning in a deployed system poses a significant challenge: llms must adapt to evolving queries and learn from user feedback over time to maintain relevance and accuracy, necessitating robust mechanisms for incorporating feedback. lastly, flexibly managing the addition and removal of candidate llms is essential; as advancements in model architectures and techniques emerge, the routing system needs to integrate new models and retire outdated ones to ensure users benefit from the latest advancements. to address these challenges, we propose mixllm, a dynamic contextual-bandit-based routing system for query-llm assignment. first, we propose a tag-enhanced embedding model by using tags generated from the instag @xcite model. these tags help improve the query representations from noises. next, we design lightweight prediction models for each llm to estimate response quality and cost. these llmspecific predictions do not require system-wide retraining when new llms are introduced. the meta decision-maker then selects the best llm for each query based on the predictions. it balances trade-offs between response quality, cost, and latency to optimize query-llm assignments. finally, mixllm benefits from continual training, allowing the system adapts to evolving queries and user feedback over time, improving the performance in real-world deployment. our extensive experiments demonstrate that mixllm effectively balances response quality, cost, and latency, achieving 97.25% of gpt-4's quality at only 24.18% of the cost. by incorporating a latency penalty, mixllm avoids congestion and high-latency issues, ensuring efficient system performance even under heavy load. additionally, we extend the routerbench dataset by incorporating the latest llama 3.1 model, showcasing the framework's scalability and adaptability. the results from online training further validate the effectiveness of the continual training approach. our contributions are as follows: ## related work the studies of selecting the most suitable llm can be categorized into non-predictive and predictive routing systems. ## conclusion we proposed mixllm, a dynamic routing system that selects the most suitable llm for each query by balancing response quality, cost, and latency. by enhancing query embeddings with tag knowledge and incorporating latency constraints, mixllm effectively addresses key challenges in real-world llm deployment. the system's adaptability, achieved through continual learning and independent prediction for each llm, ensures efficiency as queries evolve and new models are introduced. our results demonstrate that mixllm optimizes resource usage while maintaining strong performance across varying budget levels.