Bridging the Gap between Different Vocabularies for LLM Ensemble (2024)

Abstract: Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to \textbf{E} nsemble LLMs via \textbf{V} ocabulary \textbf{A} lignment (EVA). EVA bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. Specifically, we first learn mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. Subsequently, these mappings are employed to project output distributions of LLMs into a unified space, facilitating a fine-grained ensemble. Finally, we design a filtering strategy to exclude models that generate unfaithful tokens. Experimental results on commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation tasks demonstrate the superiority of our approach compared with individual LLMs and previous ensemble methods conducted on complete outputs. Further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement.

## introduction large language models (llms) have demonstrated impressive performance across various natural language processing tasks @xcite @xcite . these models, figure 1 : motivation of eva. for the problem of train travel distance, both tigerbot and chatglm provide wrong answers. ensembling over completely generated outputs cannot derive the correct answer. eva achieves correct answers by performing fine-grained ensemble at each generation step, allowing each token to benefit from the ensemble. previous studies typically concentrate on the ensemble of completely generated outputs, which involve either ranking multiple outputs to select the best one @xcite or incorporating additional fusion models to blend these outputs @xcite . therefore, these methods usually lead to ensemble outcomes confined to the space of several completely generated outputs. as shown in figure 1 , for the problem of train travel distance, both tigerbot and chatglm provide incorrect reasoning processes, resulting in wrong answers. ensembling over completely generated outputs cannot produce correct answer if all the candidate complete outputs are wrong. one potential solution to this problem involves incorporating ensembling into the generation process of llms. as indicated by @xcite , early errors in llms tend to snowball, leading to subsequent errors that might not have otherwise occurred. ensembling during generation helps prevent the generation of inaccurate tokens at each step, thereby reducing misleading cues for subsequent token generation. however, such an ensemble approach is unfeasible for llms due to vocabulary discrepancies. as illustrated in figure 2 , the three llms use distinct vocabularies, leading to different output distributions over tokens. this divergence hinders the straightforward token-level ensemble at each generation step. to tackle this issue, we propose a simple yet effective method named ensemble via vocabulary alignment (eva), facilitating the fine-grained ensemble of llms at each generation step. eva stems from a straightforward observation: although various llms have distinct vocabularies, they commonly share a significant number of overlapping tokens. by leveraging these tokens as bridges, eva can achieve vocabulary alignment. specifically, for vocabularies v q 1 , v q 2 used in llm-q 1 and llm-q 2 , we first extract embeddings of the overlapping tokens and learn a mapping matrix to project these embeddings into a shared space. subsequently, by computing similarity scores between tokens in these vocabularies, we derive the semantic projection @xmath0. this enables the projection of output distributions from llm-q 1 to llm-q 2 and generates reasonable tokens based on the fused distribution of these llms at each inference step. finally, we further enhance our approach by devising a filtering strategy capable of excluding models that generate unfaithful tokens. our method successfully overcomes the vocabulary discrepancy between different llms and facilitates fine-grained ensemble during generation. significantly, our method necessitates solely an additional projection matrix w , eliminating the necessity of extra fusion models or supervised training corpora. we evaluate our method on various nlp tasks, including commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation. experimental results demonstrate the superiority of our approach compared with individual llms and previous ensemble methods conducted on complete outputs. further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement. briefly, our contributions can be summarized as follows: however, different llms usually independently learn sentencepiece @xcite models from different training corpora, leading to different vocabularies. for instance, the vocabulary size of llama is 32,000, whereas chatglm has a vocabulary length of 125,696. such a discrepancy makes the output distributions of different models noncomparable, thereby impeding direct ensembling, as commonly practiced in conventional classification tasks. ## our method figure 3 : the rate of overlapping tokens between different llms vocabularies. the models are arranged in ascending order based on vocabulary size. each cell represents the proportion of shared tokens between the horizontal and vertical models, relative to the vocabulary size of the vertical model. considering a set of n large language models denoted as @xmath1, q 2 , • • • , q n -1 , p }, where p represents the chosen pivot model. we empirically select the model with the largest vocabulary as the pivot model p . foot_0 ## experimental results the main results on nlg tasks and reasoning tasks are shown in llms have diverse strengths and weaknesses. additionally, observing the performance of individual models on each task, we find that no models participate in every task ensemble. however, each model contributes to at least three task ensembles. this highlights the distinct knowledge possessed by each llm and emphasizes the significance of ensembling llms. output prefix he added, "we have 4-month-continuations old mice that have never had diabetus, but they have had it in the past." next token distribution 'old', 'olds', ' old', 'old', 'older', ' old', 'old', ' olds', '旧', 'olding', ... ## analysis 6.1 effect of model filtering intensity recall in section 3.2, we introduced the hyperparameter n as a way to control how strict our model filtering is. in this section, we investigate the sensitivity of our method to n. as shown in to verify our hypothesis, we conduct further experimental analysis on tasks with the highest sensitivity (gsm8k) and lowest sensitivity (machine translation). since tokens are very fine-grained units, spelling variations can directly represent semantic differences. hence, we specifically define diversity as the average edit distance between the top-n tokens and the top-1 token generated by a model. we conducted a statistical analysis on the outputs at 10,000 positions in both datasets. as depicted in figure 4 , across various top-n ranges, the edit distance for the gsm8k task consistently exceeds that of flores, confirming our hypothesis. ## related work ensemble learning is a widely adopted technique to improve performance on a given task and provide robust generalization by leveraging multiple complementary systems @xcite @xcite . existing ensemble methods can be divided into two categories: selection-based ensemble and generation-based ensemble. selection-based ensemble selection-based ensemble methods select the best output from multiple outputs. shnitzer et al. (2023) employs benchmark datasets to learn a router model responsible for selecting the best llm out of a collection of models for a given task. frugalgpt (chen et al., 2023a) calls llms sequentially until a dedicated scoring model deems the generation acceptable to effectively and efficiently leverage different llms. ravaut et al. (2022a);liu and liu (2021);liu et al. (2022) train dedicated scoring or ranking models for text summarization. farinhas et al. (2023) demonstrated that minimum bayes risk decoding is an effective ensemble method for llm-based machine translation. however, such methods are limited by the output quality of the candidate models and are unable internlm chatglm baichuan tigerbot systems 26 28 30 32 34 36 38 40 42 44 exact match 32.3 30.78 29.95 27.29 33.51 i+c 40.03 i+c+b 42.91 i+c+b+t i gsm8k single model eva to generate outputs superior to those of existing models. nevertheless, the distinctions among candidates could be quite subtle. a model's output might outperform one part compared to another model's output yet lag behind in other parts. selecting among existing answers limits the release of the complementary potential of the ensemble. fusion-based ensemble compared to selectionbased methods, fusion-based ensemble approaches bypass the limitation of existing complete outputs, often yielding superior outputs. @xcite presents a general ensemble framework utilizing a pair ranker to filter the top k optimal outputs, followed by a fusion model to merge and generate the final output. furthermore, @xcite enhances question answering by amalgamating retrieved text, while @xcite applies generative fusion methods to text summarization. however, a fusion model typically needs to have a size comparable to the base model. for instance, @xcite employs a 3b-sized model as a fusion model, significantly elevating the training and inference costs. our proposed eva conducts fine-grained ensemble at each generation step, not only obtaining new results distinct from individual model outputs but also incurring almost negligible training costs for mapping vocabularies. furthermore, our approach exhibits strong performance without the need for training on specific task datasets, demonstrating excellent generalization capabilities. ## conclusion in this paper, we propose a novel ensemble method named eva, which effectively bridges the lexical gap between different llms and facilitates finegrained ensemble at each generation step. compared to ensemble methods that select or fuse completely generated results, eva provides intermediate ensemble results to candidate models, enabling them to benefit from higher-quality output prefixes, thereby unleashing their complementary potentials. experimental results on nlg tasks and reasoning tasks demonstrate the effectiveness of our approach, which significantly improves overall performance on various natural language processing tasks.