M erge ME : Model Merging Techniques for Homogeneous and Heterogeneous M o E s (2025)

Abstract: The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging.

## introduction large language models (llms) pretrained on a wide-variety of corpora have achieved notable success in multiple tasks @xcite @xcite . with significant progress, there is increasing interest in how to continuously improve the performance of llms in new domains, including math @xcite , code @xcite , wikipedia knowledge @xcite , or legal domains @xcite . one straightforward approach is through continual pretraining (cpt) on domain-specific data, which, however, is challenging for multiple target domains, as it can cause catastrophic forgetting on previously learned tasks @xcite . an alternative approach is mixture-of-experts (moe) merging, where dense experts are first cpted in parallel for each domain and then merged into a unified moe model, usually by keeping feedforward neural network (ffn) layers separate and averaging non-ffn layers @xcite . compared with dense models of similar size, the moe model uses just a subset of parameters during inference by learning to route tokens to the top few experts, thus reducing inference costs. unlike training an moe model from scratch, moe merging offers modularity, as individual experts are domain-specialized, and is substantially less expensive, as cpt-ing experts in parallel requires less compute than training the entire moe on large datasets from the beginning @xcite . in this paper, we investigate how to effectively merge different domain expert models into a unified moe model. the current state-of-the-art (sota) moe merging approach, such as branch-train-mix (btx) @xcite assumes experts are branched from the same ancestor model and merges experts by simply unweighted averaging the non-ffn layers. however, as experts diverge in the parameter space, for example by branching from different ancestors or by training on aggressively different data, unweighted averaging may not effectively handle parameter interference such as sign conflicts @xcite . as a result, the merged moe may underperform and will require a significant amount of additional fine-tuning to recover in performance, which is both expensive and could be impractical when the experts' training data is not publicly available. furthermore, existing moe merging methods cannot be directly used to merge heterogeneous experts with different architectures, which could be the case in practice, as increasingly more experts are provided by separate teams, such as codellama @xcite and @xcite . therefore, it is still an open question how to effectively merge homogeneous and heterogeneous experts into an moe combining the benefits of each. to enable the use of diverse expert models, our work addresses the above limitations via new moe merging methodologies for both homogeneous and heterogeneous experts. in summary, our work introduces three main contributions: through extensive experiments and ablation studies across benchmarks in mathematical reasoning, programming, and general knowledge, we show that our proposed methodologies outperform previous state-of-the-art methods and extend the practical applications of moe merging. ## methodology we define our research problem as follows: given l dense expert models with parameters [θ 1 , θ 2 , . . . , θ l ], each pretrained on different domains, we aim to propose an efficient merging method to combine these dense models into an moe with parameters θ @xmath0, θ 2 , . . . , θ l ) , optimizing performance across all domains. we now present our approach for moe merging with homogeneous and heterogeneous expert models. first, for moe merging with homogeneous experts (section 3.1), we propose replacing existing averaging with more advanced merging methods to deal with parameter interference, and introduce sequence-level routing heuristics to enhance moe performance without post-merge fine-tuning. second, we introduce a novel framework for moe merging with heterogeneous experts (section 3.2), which uses projectors to unify expert inputs and outputs, and a sequence-level router. ## experiments setup and model analysis through our extensive empirical analysis, we aim to evaluate our frameworks in the settting of homogeneous experts and heterogeneous experts. ## conclusion in this paper, we propose novel methods to address challenges in the current moe merging literature. for homogeneous experts, we replace average merging in non-ffn layers with more advanced methods to reduce parameter interference. we also explore merging models into an moe without postmerge fine-tuning. for heterogeneous experts, we introduce a method using projectors and sequencelevel routing networks to combine models with different architectures. extensive empirical evaluations show that our approach significantly improves moe performance across multiple datasets.