URG : A Unified Ranking and Generation Method for Ensembling Language Models (2024)

Abstract: Prior research endeavors of the ensemble Large Language Models (LLMs) achieved great success by employing an individual language model (LM) rank before the text generation. However, the use of an individual LM ranker faces two primary challenges: (1) The time-intensive nature of the ranking process, stemming from the comparisons between models; (2) The issue of error propagation arising from the separate ranking and generation models within the framework. In order to overcome these challenges, we propose a novel ensemble framework, namely Unified Ranking and Generation (URG). URG represents an end-to-end framework that jointly ranks the outputs of LLMs and generates fine-grained fusion results, via utilizing a dedicated cross-attention-based module and noise mitigation training against irrelevant information stemming from bad ranking results. Through extensive experimentation and evaluation, we demonstrate the efficiency and effectiveness of our framework in both the ranking and generation tasks. With the close coordination of the ranking and generation modules, our end-to-end framework achieves the state-of-the-art (SOTA) performance on these tasks, and exhibits substantial enhancements to any of the ensembled models.

## introduction large language models (llms) have shown a superior performance in storing factual knowledge in their parameters @xcite @xcite @xcite . these off-the-shelf llms exhibit diverse strengths and weaknesses due to variations in data, architectures, and hyperparameters, making them * equal contribution. in recent years, the ensembling of large language models (llms) has demonstrated remarkable performance through the judicious incorporation of the diverse strengths of multiple llms. a simple approach @xcite is to concatenate all candidate results as contextual inputs to a new generative model for fusion output, but meanwhile this approach also introduces a too lengthy input for the ranker, typically, the prior ensemble approaches @xcite @xcite employed an individual ranking model ranking response outcomes from multiple llms, and then select the top-ranked candidate output as the final ensemble result feeding to a generator. the main difference in the prior works relies on the design of the mechanism in the ranking model. we select the state-ofthe-art model, llm-blender @xcite , as an example. the authors propose a coarse-grained and fine-grained mixed pipeline that firstly makes a pair-wise comparison feature matrix of n llms evaluated by a language model as the scorer, and then a generative model fuses to generate the final output from the top-k ranked outputs. however, training individual ranking models and generators faces two primary challenges. (1) the time-intensive nature of the ranking process: due to the input limitations of language models, pairwise comparisons are implemented when comparing multiple models. consequently, when dealing with n llms, the time complexity of the process becomes o(n 2 ). in addition, the individual ranking model also incurs additional memory and time costs. (2) the error propagation between ranking and generation: the ranking stage and the generation stage represent two distinct tasks with different inputs and outputs. the textual evaluation knowledge acquired by the ranking model cannot be transferred to the subsequent generator, and vice versa. the text generation objective of the generator is not directly connected to the ranking model. consequently, the framework may suffer from the introduction of poor candidates (noise) during the ranking stage, which may adversely affect the subsequent generation process. to address the aforementioned issues, we propose a novel unified ranking and generation method, abbreviated as urg. this method realizes an end-to-end framework with an encoderdecoder-based language model, encompassing both the ranking and generation modules. as depicted in figure 1 , our framework initially gathers the encoded embeddings of candidate responses from n llms, along with the encodings of the input x. subsequently, a cross-attention module is implemented to list-wise form the concatenated features composed by the pair of each n candidate and x, which has merely o(n) time complexity. this cross-attention module assigns the attention scores to each llm candidate pair, and aggregates the feature list to predict the tokens in y * . therefore, we share the parameters of this cross-attention module with that of the decoder in the generator, which also aims to predict the next token of y * based on x with the top k candidates and the history tokens predicted in y * . consequently, the decoder learns the representations of n candidates throughout both the ranking and generation stages. in the subsequent generation stage, we further introduce a kullback-leibler (kl) loss to contrastively learn the features of positive and negative llm candidates. by minimizing the kl loss, the decoder (generator) mitigates the adverse effects on the final output y * from low-quality candidates (negative). this training objective serves to mitigate the influence of noise introduced during ranking, bridging the gap between the ranking stage and the subsequent generation stage, resulting in a more robust ensemble model. we conduct a series of experiments to analyze the effectiveness and efficicenty of our proposed framework. the experimental results demonstrate that our framework achieves state-of-the-art (sota) performance in both the ranking and generation stages. furthermore, our framework exhibits a substantial improvement in running speed compared to the prior sota model, llm-blender. our contributions can be summairzed as follows: • we propose a novel framework, unified ranking and generation method (urg), which integrates the ranking and generation stages within an endto-end framework. ## related work ensemble learning is a widely employed technique that enhances accuracy and resilience in forecasting by merging predictions from multiple models @xcite @xcite . with the rapid increase in the application of large language models (llms) @xcite , the ensemble of llms has emerged as a significant topic @xcite . in light of the prior works. there are two primary approaches for ensembling llms: selection-based and generationbased methods. selection-based methods selection-based methods develop ranking models @xcite @xcite to compare candidate results generated by the multiple llms, selecting the top-1 candidate as the ensemble output. however, these methods are coarse-grained ensembles due to the inherent nature of selection and the limited solution space. as a result, they may limit the ability to harness the advantages of each candidate, potentially hindering the generation of superior outputs. generation-based methods generation-based methods employ generative models @xcite to fuse the candidate results and generate improved output as the final response. however, these methods face challenges in dealing with the surge in computational complexity and inference time with the increase of input text length @xcite @xcite @xcite @xcite . to address this issue, @xcite proposes a rank-then-generate pipeline that first adopts a pairwise ranking model to sort outputs from n llms and then utilizes a generative model to generate the final output from the k top-ranked outputs. despite shortening the length of the input text, this model still faces the problems of slow sorting speed caused by pairwise comparisons and the issue of error propagation arising from separate ranking and generation models. ## method figure 2 illustrate the overall architecture of our proposed urg framwork, which is an end-to-end model. ## conclusion in conclusion, we propose a novel framework, unified ranking and generation method (urg), which integrates the ranking and generation stages within an end-to-end framework. to enhance the robustness of the urg framework against the adverse impact of low-quality llm candidates, we introduce a kl-loss based mechanism to contrastively learn the features of positive and negative llm candidates. experiments on mixinstruct show that urg significantly reduces running time and achieves sota performance according to a variety of automatic metrics and human evaluations.