Exploring Model Kinship for Merging Large Language Models (2025)

Abstract: Model merging has become one of the key technologies for enhancing the capabilities and efficiency of Large Language Models (LLMs). The open-source community has driven model evolution by iteratively merging existing models. However, a principled understanding of the expected gains and underlying factors in model merging remains lacking. In this work, we examine model evolution through continual merging, analogous to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. With comprehensive empirical analysis, we find that there is a certain relationship between model kinship and the performance gains after model merging, which can help guide our selection of candidate models. Inspired by this, we propose a new model merging strategy: Top- k Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets. Specifically, we discover that using model kinship as a criterion can assist us in continuously performing model merging, alleviating the degradation (local optima) in model evolution, whereas model kinship can serve as a guide to escape these traps.

## introduction fine-tuning pre-trained models (ptms) for downstream tasks has become a popular practice, and has proven particularly effective for large language models (llms) @xcite @xcite @xcite . however, deploying separate fine-tuned models for each task can be resource-intensive @xcite , which drives increasing demands for multi-task learning solutions @xcite @xcite . recent studies suggest that model merging @xcite @xcite @xcite offers a viable approach for achieving multi-task objectives by integrating multiple expert models. furthermore, advances in model merging toolkits @xcite have lowered the technical barrier, allowing users with limited expertise to easily conduct merging experiments, thus leading to an evolution of llms for the community. at present, researchers have developed various powerful llms using model merging techniques @xcite . many of these models are created through a biologically inspired evolutionary process involving iterative merging, an approach that we refer to as model evolution (figure 1(a , b )). despite these successes, current merging practice faces critical limitations. progress often relies on continuous trial and extensive human expertise, with little formal guidance or standardized procedure. to address this problem, we propose an iterative model merging framework (figure 1(c )), leveraging explicit strategies to guide the direction of model evolution toward improved performance. we show that even a simple greedy strategy can outperform baseline merging approaches. however, in the later stages of both communitydriven model merging experiments and greedy evolution strategies, achieving additional gains in multi-task capability becomes increasingly challenging. to explore possible solutions, we introduce 'model kinship', a metric inspired by the concept of kinship in evolutionary biology @xcite , to inform and enhance the merging process. model kinship is designed to quantify the degree of similarity or relationship between models throughout the iterative merging process. by offering a principled framework for measuring these relationships, model kinship provides valuable insights that can refine merging strategies for model evolution. we conduct a comprehensive analysis of model merging experiments based on model kinship. we observe that the model merging process consists of two distinct stages: (1) an improving stage, where models exhibit significant performance gains, and (2) a saturation stage, where improvements diminish and eventually plateau. empirically, we find a strong correlation between model kinship and variations in average task performance, suggesting that model kinship is indicative of potential effectiveness for model merging. these findings inspire two main insights: (1) high-kinship merges can lead to performance stagnation, akin to inbreeding; (2) low-kinship merges carry greater risk but may yield larger gains and facilitate escape from local optima. inspired by this, we propose a novel continual model merging strategy: top-k greedy merging with model kinship. specifically, we find that leveraging model kinship as a criterion enables more effective model merging, helping mitigate degradation and avoid local optima during model evolution. furthermore, model kinship also proves useful as an early stopping criterion, improving efficiency of the merging process. in general, this paper mainly contains four key contributions: ## iterative model merging and strategy boost in this section, we present controlled experiments to demonstrate that iterative model merging, the process of repeatedly combining models, can help stabilize improvements in multi-task capability. moreover, we demonstrate that incorporating a selection strategy, such as top-k greedy, yields more substantial performance improvements. ## preliminary analysis of model kinship to address this limitation, we investigate whether structural signals beyond raw performance can guide model selection. we introduce model kinship, a metric capturing parameter-space similarity, and conduct a preliminary analysis to examine its correlation with merge gains in community-merged llms. this helps assess the potential of kinshipaware strategies in guiding continual merging and avoiding local optima. ## using model kinship to improve model merging building on the insights from section 4, we explore how model kinship can be leveraged to improve the model merging process. our main experiment centers on the mistral-7b model, with detailed results presented in the main text. to further evaluate the generalization of our approach, we also conduct two supplementary experiments: one based on llama-2 (see appendix b) and another on a distinct task set to test robustness across different evaluation settings. our results indicate that while greedy strategy focuses on short-term gains, it can lead to suboptimal outcomes. by integrating model kinship, we can help the strategy avoid local optima and gain further improvements. ## related work weight averaging, a widely used technique in model merging, traces its origins to @xcite . since the 2010s, weight averaging has been widely applied in deep neural networks, notably for combining checkpoints to improve training stability and performance. @xcite @xcite @xcite @xcite , leveraging task-specific information @xcite @xcite , and parallel training of large language models (llms) @xcite . discovery of linear mode connectivity (lmc) @xcite @xcite further expands the use of weight averaging in fusing fine-tuned models through averaging methods @xcite . further studies have explored optimizable weights for merging, such as fisher-merging @xcite , regmean @xcite , adamerging @xcite , mats @xcite . @xcite introduce task vectors, representing the weight difference between a fine-tuned model and its base. further research on parameter interference led to ties @xcite , which preserves important weights and reduces sign conflicts, and dare @xcite , which prevents interference by randomly dropping weights. the model breadcrumbs @xcite show that the removal of outliers in parameters can reduce noise in model merging. merging models with different initializations requires additional considerations. common methods exploit the permutation symmetry of neural networks @xcite @xcite , using alignment techniques to mitigate the interpolation barrier @xcite . while weight averaging cannot be applied to models with different architectures, it can still be used to enhance feasible fusion methods. recent work like fusechat @xcite integrates it with knowledge fusion @xcite to enable novel fusion approaches. @xcite further demonstrate that model merging can stabilize training and serve as a low-cost simulator for annealed performance during pretraining, enabling checkpoint reuse and faster validation. recently, there have been some works exploring "model evolution". tellamekala et al. (2024) propose the cold fusion method, showing that iterative fusion can create effective multi-task models. labonne (2024) develop a tool to automatically merge models on huggingface. akiba et al. (2024) introduce evolutionary model merge, employing evolutionary techniques to optimize model combinations. tang et al. ( 2025 ) present a continual model merging method that enables training-free projection-based merging of models as they arrive sequentially, improving efficiency and task retention without retraining. ## conclusion we propose iterative model merging as a framework for evolving large language models through strategic, iterative merges that yield consistent gains in generalization and task performance. to support this framework, we introduce model kinship, a metric that guides merge candidate selection and explains both performance gains and stagnation during merging. building on this, we propose top-k greedy merging with model kinship, a strategy that uses kinship to escape local optima and achieve further improvements. kinship also serves as an early stopping signal by detecting convergence and reducing redundant computation. in a broad sense, our work explores how models can achieve autonomous evolution through model merging. model merging can, to some extent, be likened to biological hybridization. biological organisms have undergone billions of years of evolution to reach their current state. however, how silicon-based intelligence, represented by llms, evolves remains an unresolved mystery. we aspire that this work offer guidance and insights for the future merging and evolution of llms.