LLM -Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (2023)

Abstract: We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.

## introduction large language models (llms) have shown impressive performance in diverse tasks, primarily due to their capacity to follow instructions and access extensive, high-quality data, showing a promising future for artificial general intelligence @xcite . however, prominent llms such as gpt-4 and palm @xcite are closed-source, restricting insights into their architectures and training data. open-source llms like pythia @xcite , llama @xcite , and flan-t5 @xcite offer a chance to fine-tune these models on custom instruction datasets, enabling the development of smaller yet efficient llms, such as alpaca, vicuna @xcite , openassistant (laion-ai, 2023), and mpt @xcite . the open-source llms exhibit diverse strengths and weaknesses due to variations in data, architectures, and hyperparameters, making them complementary to each other. figure 1 illustrates the distribution of best llms on 5,000 instructions that we collected. more ranking details can be found in sec. 5.1. although vicuna achieves the highest percentage, it ranks first in only 21.22% of the examples. furthermore, the pie chart suggests that the optimal llms for different examples can significantly vary and there is no open-source llm that dominates the competition. therefore, it is important to dynamically ensemble these llms to generate consistently better responses for each input. considering the diverse strengths and weaknesses of llms, it is crucial to develop an ensembling method that harnesses their complementary potentials, leading to improved robustness, generalization, and accuracy. by combining their unique contributions, we can alleviate biases, errors, and uncertainties in individual llms, resulting in outputs better aligned with human preferences. we introduce llm-bl e n d e r, an ensembling framework designed to achieve consistently superior performance by mixing the outputs of multiple llms. llm-bl e n d e r comprises two modules: pairranker and genfuser. initially, pairranker compares the outputs from n llms, which genfuser then fuses to generate the final output from the top k ranked outputs. existing approaches @xcite , including the reward model within instructgpt @xcite , for ranking outputs {y 1 , . . . , y n } from language models (lms) on a given input x have mostly focused on individually scoring each y i based on x, employing encoding modules in the form of s @xmath0, y i ). although this list-wise ranking objective can be powerful and efficient when candidate differences are apparent, it may not be as effective when ensembling llms. among the output candidates from llms, candidate differences can be quite subtle, as they are all produced by very sophisticated models and one may only be marginally better than another. even for humans, it can be challenging to gauge candidate quality without direct comparison. as a result, we propose a specialized pairwise comparison method, pairranker (sec. 3), to effectively discern subtle differences between candidate outputs and enhance ranking performance. in particular, we first gather the outputs from n models (e.g., the @xmath1. 1 ) for each input and subsequently create the n (n 1)/2 pairs of their outputs. we jointly encode the input x and the two candidate outputs y i and y j as input to a cross-attention encoder (e.g., roberta @xcite ), in the form of f (x, y i , y j ), to learn and determine which candidate is better. during the inference stage, we compute a matrix containing logits representing pairwise comparison results. given this matrix, we can infer a ranking of the n outputs for the given input x. subsequently, we can employ the top-ranked candidate from pairranker for each input as the final result. hence, this approach does not rely on a single model for all examples; instead, pair-ranker selects the best model for each example by comprehensively comparing all candidate pairs. nonetheless, this approach may constrain the potential to generate even better outputs than the existing candidates. to investigate this possibility, we introduce the genfuser (sec. 4) module to fuse the top k of the n ranked candidates and generate an improved output for end-users. our goal is to capitalize on the strengths of the top k selected candidates while mitigating their weaknesses. to assess the effectiveness of llm ensembling methods, we introduce a benchmark dataset called mixinstruct (sec. 2.2). in this dataset, we use @xmath2. the dataset comprises 100k training examples and 5k validation examples for training a candidate ranking module like our pairranker, and 5k test examples with oracle comparisons for automatic evaluation. in section 5, our empirical results on the mixinstruct benchmark reveal that the llm-bl e n d e r framework significantly boosts overall performance by ensembling llms. the selections made by pairranker outperform any fixed individual llm models, as indicated by superior performance in both reference-based metrics and gpt-rank. by leveraging the top selections from pairranker, genfuser further enhances response quality through effective fusion into the final output. llm-bl e n d e r achieves the highest scores in terms of both conventional metrics (i.e., bertscore, bartscore, bluert) and chatgpt-based ranking. the average rank of llm-bl e n d e r stands at 3.2 among the 12 methods, which is considerably better than the best llm's rank of 3.90. moreover, llm-bl e n d e r's output ranks in the top 3 for 68.59% of examples, while viccuna only reaches 52.88%. we believe llm-bl e n d e r and our findings would benefit both practitioners and researchers for deploying and studying llms with ensemble learning. ## preliminaries we first provide the problem formulation and two common types of ensembling methods. next, we for each input x from users, we employ n different llms to get output candidates. then, we pair all candidates and concatenate them with the input before feeding them to pairranker, producing a matrix as comparison results. by aggregating the results in the matrix, we can then rank all candidates and take the top k of them for generative fusion. the genfuser module concatenates the input x with the k top-ranked candidates as input and generate the final output ŷ. ## pairranker: pairwise ranking in this section, we introduce three baseline methods for ranking the candidates in y in sec. 3.1 and present the proposed pairranker method. ## genfuser: generative fusion the effectiveness of pairranker is constrained by the quality of selections from the candidate pool y. we hypothesize that by merging multiple top-ranked candidates, we can overcome this to accomplish this, we present genfuser, a seq2seq approach for fusing a set of candidates conditioned on the input instruction to generate an enhanced output. specifically, we concatenate the input and k candidates sequentially using separator tokens, such @xmath3, and fine-tune a t5-like model to learn to generate y. in practice, we employ flan-t5-xl @xcite , which has 3b parameters, due to its superior performance and relatively smaller size. ## conclusion & future directions in this paper, we formulated the motivation to exploit the diverse strengths and weaknesses of open-source large language models (llms), aiming to create an ensembling framework that leverages their complementary capabilities to generate consistently superior results on various instructionfollowing tasks. by dynamically ensembling llms, we aimed to reduce biases, errors, and uncertainties in individual models, yielding outputs better aligned with human feedback. our major contributions are as follows: future directions. potential future directions include extending the llm-bl e n d e r framework to more types of models or even non-text modalities, developing more sophisticated ranking and fusion techniques, and investigating the transferability of our ensembling approach to other domains and tasks. additionally, exploring ways to minimize computational overhead and incorporating active learning strategies for rapid adaptation to new specialized domains and data sources represent fruitful areas for further research. overall, our work underscores the value of combining the unique contributions of multiple models.