Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models (2024)

Abstract: The complementary potential of Large Language Models (LLM) assumes off-the-shelf LLMs have heterogeneous expertise in a wide range of domains and tasks so that an ensemble of LLMs can achieve consistently better performance. Existing ensemble methods for LLMs mainly focus on reward model ranking of outputs, leading to significant computation overhead. To combat this issue, we revisit the complementary potential of LLMs and further elaborate on it by mining latent expertise with off-the-shelf reward models. We propose ZOOTER, a reward-guided routing method distilling rewards on training queries to train a routing function, which can precisely distribute each query to the LLM with expertise about it. We also integrate a tag-based label enhancement to mitigate noise from uncertainty when using rewards as silver supervision. ZOOTER shows computation efficiency in inference as it only introduces minor computation overhead of a routing function compared with reward model ranking methods. We evaluate ZOOTER on a comprehensive benchmark collection with 26 subsets in different domains and tasks. ZOOTER outperforms the best single model on average and ranks first on 44% of tasks, even surpassing multiple reward model ranking methods.

## introduction large language models (llms) aligned with human preference rapidly emerge and are almost daily released @xcite @xcite . these off-the-shelf llms are further finetuned or aligned with human preference to be generalists @xcite or specialists @xcite @xcite for solving versatile tasks. it is worth noticing that llms are pretrained and aligned with various data, leading to diverse strengths and weaknesses in versatile downstream tasks @xcite . therefore, the ensemble of llms harnesses the complementary potential among them and may achieve better performance than a single best-on-average model across diverse tasks. one of the key challenges in the llm ensemble is computation efficiency due to the large parameter size of existing llms. previous research @xcite provides solid methods to merge generation outputs of llms as an ensemble. such methods require tremendous inference cost that makes it unscalable and thus not competitive to the best-on-average model under low-resource scenarios. to efficiently assemble off-the-shelf llms, we first dive deeper into the considerably straightforward but still understudied assumption: off-the-shelf aligned llms, even for those aligned as "generalists", have heterogeneous expertise in a wide range of domains and topics. however, analyzing the expertise of an llm is also challenged as the latent expertise of llms is highly related to the pretrained and alignment data, which is very vague and inaccessible even for popular open-source llms such as llama-2-chat @xcite and wizardlm @xcite . if this assumption strongly holds, off-the-shelf llms can be assembled efficiently by assigning queries to the model that is proficient in the query without additional inference costs on each model. such an efficient routing strategy only requires inference cost for a single model for each query and the overhead cost of a much smaller query router. however, probing the detailed expertise of off-theshelf llms and generating supervision for training routers also require annotations. developing a data-efficient training method for routing queries is significantly understudied. to combat these issues, we propose zooter, a reward-guided query routing method for efficiently assembling off-the-shelf llms. zooter obtains and enhances silver supervision from existing reward models (rm) for query router training and distributes queries in advance to "experts". as shown in fig. 1 , the reward distribution implies the oracle judgments and reveals a latent expertise between llms. and zooter captures the expertise from reward distributions and provides query distribution during inference. specifically, we first conduct a comprehensive study involving four groups of benchmarks across more than 26 subsets in various domains and tasks. we investigate six widely used open-source llms and show the complementary potential of such wide-range downstream tasks by aggregating them via reward model ranking. we then collect a diverse training query set and distill rewards of model expertise as indirect supervision for training an llm router and develop tag-based label enhancement to overcome the shortage of such silver labels from reward models further. with comprehensive experiments, we show zooter can benefit from rm silver supervision to learn the latent expertise among llms and conduct efficient routing for the model ensemble. our contributions are mainly three-fold: the quick emergence of aligned llms motivates us to develop routing techniques to maximize the strengths of ensemble llms. in this work, we focus on assembling aligned llms, such as llama-2-chat @xcite , wizardlm @xcite , vicuna @xcite , and so on. and we evaluate them on a wide range of alignment evaluation tasks. large language model ensemble. the ensemble of llms is an emerging topic due to the explosion of open-source llms. llm ensemble aims to merge off-the-shelf llms to perform better consistently across diverse downstream tasks. few works explore the complementary potential assumption of llms and how to assemble llms with it. jiang et al. (2023) presents an ensembling framework consisting of a pair ranker and a generation fuser. chen et al. (2023) sequentially infers off-the-shelf llms and stops until the response meets a sufficient quality. wang et al. (2023b) proposes a fusing-of-experts problem that fuses outputs of expert models with complementary knowledge of the data distribution and formulates it as supervised learning. shnitzer et al. (2023) show the utility and limitations of learning model routers from various benchmark datasets. although these works all focus on reward ranking or routing strategies to assemble llms, zooter distinguishes from these concurrent works in two aspects. first, our concurrent works require output generations or the forward process to get prompt representations of all candidates, leading to significant computation overhead. zooter infers model expertise by distilling rewards on a predefined training query set to avoid such inference overhead. then, all these works are developed and evaluated on a set of benchmarks. at the same time, zooter can be developed with only queries without golden responses, and zooter aims for more diverse alignment tasks. therefore, zooter stands out for its efficiency in data and computation. we also evaluate zooter on more diverse alignment tasks to comprehensively examine the complementary potential of llms. reward model guided generation. reward models in the context of large language models are commonly used to improve alignment performance by reinforcement learning @xcite or preference learning @xcite @xcite . reward models can also improve the performance during the generation phase. the math reasoning ability of language models can be improved by using reward models ranking multiple generated reasoning paths @xcite @xcite . @xcite uses reward models to formulate rewardguided decoding. inspired by these successful applications of reward models in alignment, zooter also takes advantage of off-the-shelf reward models to investigate the latent expertise of llms. ## methods we first revisit the complementary potential of llms ( §3.1) and then introduce zooter as an efficient llm ensemble method ( §3.2). ## experiments in this section, we report experimental setup ( §4.1), main results ( §4.2), and analysis about zooter ( §4.3).