R outer E val: A Comprehensive Benchmark for Routing LLM s to Explore Model-level Scaling Up in LLM s (2025)

Abstract: Routing large language models (LLMs) is a new paradigm that uses a router to recommend the best LLM from a pool of candidates for a given input. In this paper, our comprehensive analysis with more than 8,500 LLMs reveals a novel model-level scaling up phenomenon in Routing LLMs, i.e., a capable router can significantly enhance the performance of this paradigm as the number of candidates increases. This improvement can even surpass the performance of the best single model in the pool and many existing strong LLMs, confirming it a highly promising paradigm. However, the lack of comprehensive and open-source benchmarks for Routing LLMs has hindered the development of routers. In this paper, we introduce RouterEval, a benchmark tailored for router research, which includes over 200,000,000 performance records for 12 popular LLM evaluations across various areas such as commonsense reasoning, semantic understanding, etc., based on over 8,500 various LLMs. Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement.

## introduction routing llms @xcite @xcite @xcite aims to establish an efficient router capable of selecting an llm from a pool of candidates with varying abilities to effectively handle a given input, in order to achieve specific goals such as high overall accuracy, low computational cost, or minimal hallucination, as shown in fig. 1 . since this paradigm only involves input assignment, it is naturally compatible with heterogeneous llms of different structures, as well as most model enhancement methods, including fine-tuning @xcite , mixture-of-experts @xcite ., input 1 input 2 router llm pool what's the chance of this paper being accepted? how can i modify this paper to make it the best paper? no way 50% figure 1 : the overview of routing llms. for each given input, the router distributes it to the appropriate llm to achieve specific objectives, such as high accuracy, low computational cost, reduced hallucinations, etc. we find that routing llms is a promising paradigm for llms to achieve model-level scaling up. ## preliminary notation. assume there is a llm pool containing m llms {ℓ i } m @xmath0. for a given input s j , its representation κ(s j ) can be obtained using an encoder κ. based on the performance records of these llms on s j , we can obtain a m-dimensional one-hot selection vector v j , where the dimension marked as 1 indicates that the corresponding llm indexed by that dimension achieves the dominant performance among {ℓ i } m @xmath1, otherwise it is marked as 0. the vector v j indicates the optimal choice for the router on the input s j . specifically, taking @xmath2, if the performance metric is correctness or incorrectness, the dimensions of v 0 corresponding to the llms that provide correct answers can be simultaneously marked as 1, such as v 0 = [1, 1, 0] or even v 0 = [1, 1, 1], since these llms can all be considered as the optimal choices for the router. moreover, if the metric is continuous, the index of the llm whose performance is within 95% of the optimal performance can be marked as 1, otherwise, it is marked as 0. router. in practice, the task of the router r θ with learnable parameters θ is to establish the following mapping using the given input set {s j } n @xmath3, and possible external data d as the training set: if there is no external data, then @xmath4. after training, for an unseen input s ′ , the trained router is required to generate a selection vector v ′ , and based on this vector, determine the optimal llm from the llm pool for the current input. thus, the acquisition of the router can be regarded as a classic classification problem @xcite @xcite @xcite . given that the development of routers is still in its nascent stage, in this paper, we prioritize the performance corresponding to each benchmark when selecting llms, without considering factors such as computational cost, hallucination rate, etc. see section 6 for more analysis. ## model-level scaling up phenomenon in this section, we comprehensively explore the relationship between the number of llm candidates and the final performance under different router capabilities, to demonstrate the immense potential of routing llms paradigm. specifically, we consider the four well-known llm benchmarks arc @xcite , mmmu-pro @xcite , math lvl 5 @xcite , truthfulqa @xcite , and collect performance records of thousands of llms on these benchmarks. using collected performance records, we can construct an oracle router r o , which can select the optimal llm to process a given input s from any m llm candidates. furthermore, we can construct other routers with different capabilities by defining r o (p) as r o (p) = r o , with probability p, ω m , with probability 1p, subsequently, for a given m, we repeatedly sample m llms from the llm pool with equal probability, 100 times, forming 100 candidate subsets each containing m llm candidates. we then calculate the average performance of different routers r o (p) on these subsets. the statistical results are shown in fig. 2 , and we have following finding: as the number of candidates increases, most routers r o (p) can rapidly enhance performance, especially when @xmath5.5. this implies that, given sufficiently well-constructed routers, routing llm is an effective paradigm for scaling up llms at the modellevel. in fact, this phenomenon aligns with the classical neural scaling law @xcite , as the increase in the number of candidates can be regarded as an implicit increase in the number of parameters, which is manifested in a sparse manner under the routing llm paradigm. in fact, the vast majority of llms discussed in this paper are open-source and can be deployed locally (see appendix b for details). the individual performance of these models is not particularly remarkable and generally falls significantly short of mainstream llms. however, as shown in fig. 2 , even relatively weak candidates can achieve complementary performance among multiple heterogeneous models under the routing llms paradigm. they can obtain fine-grained division of labor for each input and outperform mainstream llms. this shows that relatively weak candidates can also be promising for obtaining high performance within this paradigm. see more details in section 4.2. in fig. 2 , we not only observe that a sufficiently large number of candidates can achieve excellent performance, but also note that even with only 3 ∼ 10 candidates, the performance can obtain strong performance, and even surpass that of the strong reference model. therefore, when only moderately good performance is required, a small number of candidates is sufficient, which is highly advantageous for users with limited resources to adopt the routing llms paradigm. coupled with the aforementioned observation that weak candidates also hold great potential, the routing llms paradigm demonstrates significant utility and applicability. ## the construction of routereval in this section, we elaborate on the construction of our comprehensive benchmark for routing llms, named routereval. we collected over 200,000,000 performance records for 12 popular llm evaluations spanning areas such as knowledge-based q&a, commonsense reasoning, semantic understanding, and instruction following, based on more than 8,500 llms. the benchmarks @xcite @xcite @xcite @xcite @xcite @xcite involved include arc, hellaswag, mmlu, truthfulqa, winogrande, gsm8k, ifeval, bbh, gpqa, musr, math lvl 5, and mmlu-pro. see appendix b, a and d for more details. ## experiments in this section, we present the performance of various existing routers on our proposed routereval. specifically, in these experiments, we do not consider the extra data mentioned in section 4.3, since the utilization of such data is highly diverse and can be considered from multiple perspectives. ## analysis in this section, we conduct a more comprehensive analysis of the proposed routereval. in fact, several existing paradigms in other fields are related to routing llms. this section aims to analyze their relationships and differences. see related works in appendix c for more details. recommender system. in fact, routing llms are a specialized type of recommender system (rs), where the input can be seen as the user in the rs, llm candidates as the items to be recommended, and the performance record as the interaction history between items and users. given the input, the router needs to recommend an appropriate llm to achieve various objectives, such as high accuracy, low computational cost, or minimal hallucination. however, compared to traditional rs, routing llms have very limited "user information" that can be collected, and labeling this data is challenging. most of the data is private and rarely open-source. the over 200,000,000 performance records we have organized and open-sourced represent only a small step toward building a router with strong recommendation capabilities. llms ensemble. for a given input, the llms ensemble paradigm focuses more on requiring all llm candidates to perform inference, then aggregating and organizing their results, with the final output achieved through strategies such as majority voting. in contrast, the routing llms paradigm performs an efficient assignment before the candidates' inference, making it a more computationally efficient approach. of course, since different paradigms can borrow from each other, there is still some technical overlap between these two paradigms in many improvement methods. ## conclusion in this paper, we comprehensively explored the potential of the routing llms paradigm through extensive experiments and identified the scaling-up phenomenon of llms at the model level. furthermore, given that the development of routers is hindered by the lack of comprehensive benchmarks, we introduce routereval, a benchmark based on 200 million performance records across 12 llm evaluations. the evaluations reveal that existing routing methods still have room for improvement.