Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing (2024)

Abstract: With the rapid development of LLMs, it is natural to ask how to harness their capabilities efficiently. In this paper, we explore whether it is feasible to direct each input query to a single most suitable LLM. To this end, we propose LLM routing for challenging reasoning tasks. Our extensive experiments suggest that such routing shows promise but is not feasible in all scenarios, so more robust approaches should be investigated to fill this gap.

## introduction large language models (llms) demonstrate remarkable capabilities in many natural language generation and understanding tasks @xcite @xcite . at the same time, @xcite show that no single open-source llm outperforms all others across different benchmarks and datasets, as various llms may exhibit different domain expertise @xcite . experiments towards predicting model behavior @xcite also suggest that particular aspects of input prompts can affect different llms in different ways. it is, therefore, reasonable to investigate whether the capabilities of different llms can be harnessed to achieve better results more efficiently. recent findings suggest that performance can be improved with ensembling @xcite @xcite and collaborative frameworks @xcite . however, the research in this area is still in the early stages, with a number of open research questions. in this work, we propose llm routing, which investigates whether directing an input prompt to the most suitable single llm can lead to better performance than what can be achieved with individual llms while maintaining a reasonable (e.g., single llm) latency. with the rise of larger and more capable models in nlp and the wider field of ml, the research on sparse expert models has also extended. this class of models includes mixture-of-experts @xcite @xcite , switch-transformers @xcite , and routing networks @xcite , among other models. 2 approaches to building these sparse models vary along several dimensions: (i) how the optimal parameter subset(s) or model-pool candidates are identified for each input (e.g., feature-based or deep-encoder-based classification), (ii) whether the subset selection involves pre-training the candidate models or model layers (e.g., mixtral @xcite ), which can incur significant training compute and data costs, (iii) how many experts are selected for each input (e.g., hybridllm @xcite selects only the single best, whereas @xcite selects the top-k), and (iv) whether the approach also aims to improve the response quality or overall performance beyond that of any single candidate model. in this context, our paper aims to build and analyze a sparse llm routing model that selects the single best llm (from a pool of at least two llms) for each input query. the proposed router only requires fine-tuning of a relatively small pre-trained transformer encoder model on the data without the need for pre-training or fine-tuning the llms. given that llms frequently face challenges with reasoning and planning tasks @xcite , we focus on two well-established reasoning task benchmarks. we empirically investigate the feasibility of building llm routing model capable of selecting the most suitable llm for each input from a pool of diverse llms. the routing is grounded on responses generated by llms. the contributions and key findings of this work are as follows: (1) we propose an llm routing model, which directs input queries to the most suitable single llm. (2) we explore two different types of approaches for llm routing, treating it as a classification and a clustering task. (3) we conduct experiments with 7 open-source llms and on two challenging reasoning benchmarks (gsm8k and mmlu). (4) we introduce theoretical upper bounds for two scenarios: (i) highest possible performance achieved jointly with all llms (i.e., oracle), and (ii) highest performance achieved with the proposed routing model. (5) our findings indicate that theoretical upper bounds of the routing model are higher than individual model performance, however, the proposed model developed in practice is unable to achieve those scores. specifically, the performance of the routing model is better than that of the weak llms but is similar to or slightly lower than that of the top-performing llms, which may be due to the small size of the training data. despite the somewhat negative results, we believe this study demonstrates the feasibility of modeling llm routing and contributes to new research directions on efficient usage of llms, which can benefit researchers and practitioners. ## methodology we present an overview of the proposed workflow in figure 1 . below, we describe our approaches to llm sampling and llm routing. split/criteria gsm8k mmlu training 6,816 13,757 validation 359 285 test 1,319 1,530 #examples for few-shot cot 5 5 ## experimental setup llm routing baseline models the following baseline models are included for comparison: ## results and discussion in