S elect LLM : Query-Aware Efficient Selection Algorithm for Large Language Models (2025)

Abstract: Large language models (LLMs) have been widely adopted due to their remarkable performance across various applications, driving the accelerated development of a large number of diverse models. However, these individual LLMs show limitations in generalization and performance on complex tasks due to inherent training biases, model size constraints, and the quality or diversity of pre-training datasets. A promising direction is to efficiently harness the diverse capabilities of LLMs to overcome these individual limitations. To address these limitations, we introduce a novel LLM selection algorithm called SelectLLM, which efficiently directs input queries to the most suitable subset of LLMs from a large pool, ensuring that the selected models collectively provide accurate responses. SelectLLM employs a multi-label classifier and policy based on the classifier’s predictions and confidence scores in selecting an optimal, query-aware, and lightweight subset of LLMs. Our findings indicate that the proposed model outperforms existing ensemble-based baselines and achieves competitive performance with similarly sized top-performing LLMs while maintaining efficiency. Specifically, it achieves a huge reduction in inference latency on two challenging reasoning benchmarks: 13% on GSM8K and 70% on MMLU, compared to the top-performing baseline. Also, we establish a theoretical upper bound by an Oracle with LLMs and perform an in-depth linguistic analysis to understand the performance gap between the Oracle and SelectLLM.

## introduction in recent years, large language models (llms) have demonstrated remarkable capabilities in solving a wide range of core nlp tasks @xcite . despite these advances, existing llms still struggle with complex tasks such as factually-grounded reasoning and planning @xcite @xcite . moreover, the wide range of llms available seem to exhibit diverse capabilities @xcite , resulting in no single (especially open-source) llm being effective across all benchmarks and datasets. aligning with recent research trends @xcite , we focus on natural language understanding tasks grounded in reasoning, specifically those with discrete output values, and do not consider general natural language generation tasks. although newer and more powerful models are constantly introduced, an alternative and costeffective approach involves harnessing the diverse capabilities of existing models to improve the overall response quality using ensembling @xcite @xcite and collaborative frameworks @xcite . however, these approaches often require access to the responses from all models in the pool to choose the optimal response(s), which greatly increases the overall computational cost for such ensembles. the individual llms exhibit diverse capabilities, i.e., not all models may be suited for all kinds of tasks. fig. 1 reports the accuracy of a llm model pool (spanning up to 7 diverse llms) on two challenging reasoning benchmarks -gsm8k @xcite and mmlu @xcite . as the plot demonstrates, utilizing more llms initially improves performance, which is supported by previous research towards employing more llms and more responses per model @xcite . however, note that using more (or even all) models in the pool does not necessarily result in the best scores overall. thus, selectively abstaining from querying unsuitable llms for a given task may help improve the overall response quality of such ensembles. additionally, such an approach would implicitly save computational resources by accessing fewer models per query. in this paper, we propose the novel se-lectllm algorithm to explore this idea. our approach first employs a multi-label classifier to learn the llm-specific capabilities using a dataset of diverse queries. when running inference for a unseen query, this knowledge is utilized to predict confidence scores for each llm in the model pool, reflecting their likelihood of successfully solving the task. next, we develop various selection policies to determine the optimal subset of llms for each query based on these confidence scores and predictions. additionally, we establish a theoretical oracle model's upper bound that can be achieved collectively by all llms and perform a qualitative and quantitative linguistic analysis of the inputs to understand the performance gap between the oracle and selectllm. the contributions of our work are as follows: ## problem setting we propose an ensembling-based llm inference algorithm -selectllm -to efficiently select the most suitable query-aware few llms from a large pool of available llms. the algorithm harnesses the diverse capabilities of different llms and selects a subset of models for the input query, jointly leading to the correct response, and this selection of a small subset leads to the reduction in latency. formally, for a given set of input queries @xmath0, q 2 , . . . , q n } and a pool of llms @xmath1, l 2 , . . . , l k }, the objective is to learn a selection model m that selects a subset of llms l s ⊆ l, which jointly produce the correct answer for a given input query q @xmath2, such that the cumulative latency satisfies latency(l s ) < latency(l). in the best-case scenario, a query q i is processed by a single llm, while in the worst-case scenario, q i is processed by all llms in l. ## results and discussion the effect of different policies with the se-lectllm algorithm: it can be observed that the labelledmaxconf policy yields the lowest performance on the gsm8k dataset. this suboptimal performance may be attributed to the policy's dependence on both the mlc predictions and confidence scores, wherein the classifier predominantly assigns the label metamath-7b-lm. given that metamath-7b-lm is a specialized model for mathematics and 88% of the sldata training data is associated with this label, this reliance skews the predictions. however, this effect is minimal for the mmlu dataset, where the label distribution across llms is more balanced. this limitation is addressed by the maxconf and weighted-maxconf, which relax the constraint on mlc label prediction and only operate on confidence scores. this allows models to incorporate other llms and push the performance, particularly for the gsm8k data. mathematical tuning in weight-edmaxconf allows policies to select llms more effectively and improve the scores. overall, the weightedmaxconf policy emerged as the best performing, with a slight edge over its closest competitor, the maxconf policy. selectllm vs. baseline models: the proposed selectllm model, utilizing the weight-edmaxconf policy, demonstrates superior performance compared to the random, all llms, and llm-blender baselines. additionally, se-lectllm maintains competitive accuracy relative to the top-s llms baseline while achieving significantly lower latency. specifically, it reduces latency by 13% for the gsm8k dataset and by 70% for the mmlu dataset compared to all baselines (except random). this substantial efficiency gain underscores the effectiveness of selectllm in balancing performance and operational efficiency. ## conclusions and future directions in this work, we introduce an efficient and novel llm selection algorithm, selectllm, to nav-